DOCUMENT RESUME 



CE 079 649 

Moore, Richard W. ; Blake, Daniel R.; Phillips, G. Michael; 
McConaughy, Daniel; von Haam, Amy Cheung 
ETP at Work: An Evaluation of 1995-96 ETP Projects. 
California State Univ. , Northridge . 

California State Employment Training Panel, Sacramento. 
2000-01-27 

83p . ; For an evaluation of ETP 1 s impact on trainees, 
companies, and the state's economy, see CE 079 650. 

Reports - Evaluative (142) 

MF01/PC04 Plus Postage. 

Case Studies; Consortia; Contracts; * Education Work 
Relationship; * Employer Attitudes; Field Studies; Models; 
Outcomes of Education; Productivity; Program Effectiveness; 
Questionnaires; Reinforcement; *School Business 
Relationship; *State Programs; State Surveys; Statewide 
Planning; *Student Attitudes 
*California Employment Training Panel 

Qualitative methods were used to analyze a sample of 23 
representative California Employment Training Panel (ETP) projects to 
determine how ETP training affected companies. Data were collected through 
interviews, focus groups, surveys, observations, and document analysis. Among 
the factors found to effect quality of training design were: selection of 
workers for training; selection of skills to be taught; timing of training in 
relation to other changes in the workplace; plans to institutionalize 
training; and amount of planning invested in reinforcing changed behavior 
after training. On average, trainees gave their training a rating of 3.11 on 
a 4-point scale. More than two-thirds of trainees reported being able to 
learn most or all of what was taught. The following five categories of 
management intervention guided the effect of training on company quality and 
productivity: (1) shaping the meaning of training; (2) creating opportunities 

to use skills; (3) creating rewards and incentives; (4) establishing levels 
of participation; and (5) institutionalizing training. Stand-alone employer 
contracts, consortia projects, and training agency projects teaching specific 
and generic skills were effective because they made it possible to customize 
training to trainee and employer needs. (Contains 17 references and 12 
tables/figures. The trainee and supervisor evaluation questionnaires are 
appended.) (MN) 



ED 437 561 

AUTHOR 

TITLE 

INSTITUTION 
SPONS AGENCY 
PUB DATE 
NOTE 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 

ABSTRACT 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 




ETP At Work: 

An Evaluation of 1995-96 ETP Projects 



or- 

cn 

O 




By: 



Richard W. Moore, Daniel R. Blake 

G. Michael Phillips, Daniel McConaughy 

and Amy Cheung von Haam 

College of Business Administration and Economics 

California State University, Northridge 



U.S. DEPARTMENT OF EDUCATION 
Offtcp of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

This document has been reproduced as 
received from the person or organization 
originating it. 

Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Submitted To: 

California Employment Training Panel 
Sacramento, California 

PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

January 27, 2000 

TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



BESTCOPY AVAILABLE 



2 



ETP At Work: 

An Evaluation of California’s 1995-96 ETP Projects 

Executive Summary 



Overview 

Previous evaluation studies of the California Employment Training Panel (ETP) have 
focused on individual trainee’s earnings and employment. While these studies revealed that 
ETP had a positive impact on the trainees, they did not examine how ETP training leads to 
increased earnings or reduced unemployment. Similarly, earlier studies assumed that if 
trainees’ earnings increased then company productivity would increase as well. This study 
explored new territory. In this study we examined the dynamics of how ETP training 
affected companies. We investigated why some projects seemed to have a major impact on 
companies and why other projects had a much smaller impact. 

Approach 



Answering these types of questions requires a qualitative rather than a quantitative approach. 
We began our analysis by choosing 23 representative ETP projects that had ended in 1995- 
96. Some were employer contracts, some were consortia projects, and some were training 
agency projects. The sampled projects represented employers of different sizes, in different 
industries and in different regions of the state. 

At each project site we used a variety of research methods including: 

• Interviews: structured interviews were conducted with managers and supervisors, 

• Focus groups: focus groups were conducted with trainees and supervisors of 
trainees, 

• Evaluation questionnaires: trainees and their supervisors completed 
questionnaires that rated the quality of training, the amount they learned and the 
impact of the training on their productivity, 

• Observations: observations were made of trainees at work on production, 

• Document analysis: researchers examined relevant documents including the 
training curriculum, ETP’s project file, company productivity records, quality 
team meeting notes, and other records. 

Qualitative methods like these generate rich data and lead to important insights, but due to 
the relatively small number of projects studied the results cannot necessarily be generalized 
to all ETP projects. 
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Results 



The Model 

Qualitative fieldwork allowed us to probe the complexity of the ETP training projects we 
studied. The results of our extensive analysis can be summarized in a model (displayed 
below) that captures conceptually the basic dynamics of how ETP training programs do or do 
not lead to increased company performance. 



Figure 1: Training Impact Model 
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There are two key points about the model: it is sequential and multiplicative. It is sequential 
in that each element appears in the chronological order in which they occur in a training 
program. Before training, a company’s production has some existing relationship to the 
industry’s best practice. Obviously, the planning of training precedes the actual training. 
Next, the reinforcement of training must come after the training. Finally, “Value of Potential 
Gains Realized” can be conceptualized as the dollar value of improved productivity. 

The model is multiplicative in that the quality of each component amplifies or diminishes 
what has come before. For example, relatively poor training with strong reinforcement of 
what was learned will still have a significant impact. Conversely, exemplary classroom 
training with little management reinforcement will have little impact. Also, if any of the 
factors are completely absent, a value of zero, there will be no impact. 

The model quickly illustrates a few lessons from the fieldwork. First, the companies’ 
potential gains from training vary substantially. Companies far down the proverbial learning 
curve actually have greater potential benefits from training than better-run companies which 
have already achieved a substantial portion of the potential productivity gains. Second, high 
quality training contributes to the potential gains from training but does not guarantee 
productivity increases. Finally, management behavior controls the degree to which potential 
gains are actually achieved. The model indicates that if there is no management 
reinforcement (a value of 0), there will be no change in productivity. 

We summarize the results of fieldwork under each component of the model. 

• Potential Gains 
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In our field visits we were struck by the wide variation in the potential of companies to gain 
from ETP training. The basic paradox is this: companies with strong management are likely 
to have achieved a large proportion of the potential gains from training. These well-run 
companies have already improved their production process and thus have less potential for 
benefiting from ETP training, yet these same companies may be best able to obtain and 
execute an ETP project because they do have strong management. Conversely, companies 
far below the best practice standard for their industry, due to weak management or other 
problems, have the most to gain; but they are likely to have the most problems executing an 
ETP funded program and supporting changes in the production system. 

• Training Design 

We found that the quality of training design was affected by how well the following activities 
were executed: 

^ selecting workers for training, 

S selecting skills to be taught, 

S selecting level at which skills will be taught, 

^ timing training in relation to other changes such as introduction of technology, quality 
teams, or incentive pay, 

S plans to institutionalize training, 

S the decision to use in-house or contract trainers or both, and 
S the amount of planning invested in reinforcing changed behavior after the training. 

• Quality of Training Delivered 

We asked trainees to rate the quality of the ETP training they received on a standard 
evaluation questionnaire. The results show that trainees rated the overall quality of training 
as slightly better than “good,” giving it a rating of 3. 1 1 on a 4-point scale. The trainees rated 
nine aspects of training. The highest rated aspects were the quality of instructors and their 
ability to hold the trainee’s interest, along with the clarity of the training objectives and the 
quality of instructional materials. Lower rated aspects included the degree to which the 
training was customized to the company, the effectiveness of SOST (structured on-site 
training), the time dedicated to each topic, and the level of training. 

A second key indicator of quality is whether or not trainees learned what was taught in 
training. Again the results show that overall ETP training was successful; over two thirds of 
the trainees reported that they were able to leant “most” or “everything” of what was taught. 
Less than two percent of trainees said they learned none of what was taught. 

The fieldwork convinced us that the customization of training was critical to creating 
effective ETP training. Customization needs to occur on three levels. First, the levels at 
which topics are taught need to be customized to the level of trainees. Second, the training 
needs to be customized to the unique processes used by the company. Finally, training needs 
to be in tune with the company’s culture. 
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We found that the quality of SOST varied widely across the projects we visited. SOST is a 
powerful instructional approach when applied correctly, but all too often we found that SOST 
activities had limited relevance to training, were poorly supervised and contributed little to 
the effectiveness of training. 

• Management Reinforcement 

Our model shows that effective reinforcement of training leads to significant impact, while 
no reinforcement negates the impact of even good training. Based on our fieldwork we 
identified five categories of management intervention, which shape the impact of training on 
the company’s quality and productivity. They are: 

S shaping the meaning of training, 

S creating opportunities to use skills, 

S creating rewards and incentives, 

^ establishing levels of participation, and 
S institutionalizing training. 

• Impact of Training 

We began our analysis of the impact of the projects studied by looking at the impact on 
individual trainees; specifically we analyzed whether they were using the skills they learned, 
and if their individual productivity had increased. Ninety-five percent of the trainees said 
they used the skills they learned at least occasionally and over half said they used them daily. 
Two thirds of trainees reported a substantial or major increase in their productivity due to 
training. In all cases, there were significant differences based on the type of training; overall 
“production techniques” and “TQM” training appeared to have the biggest impact. 

Training also seemed to improve the work environment. Most employees reported that after 
training they felt more motivated and involved at work and in general had a more positive 
attitude towards the company. 

Using data from interviews, we attempted to get evidence of ETP training’s impact on each 
company’s overall productivity, using data such as scrap rates, quality measures, and labor 
productivity measures. From the twenty-three cases we drew a few conclusions. Overall 
ETP training appeared to have substantial positive impact on company productivity. 

Employer contracts appeared to have a greater impact on company productivity because they 
were more customized to the company’s needs than consortia or training agency projects. 

We also found that training in TQM and related techniques tended to trigger a series of 
innovations that yielded substantial increases in productivity and that this was true in both 
low tech and high tech companies. We also found that companies that used in-house trainers 
rather than contract trainers tended to have greater productivity increases. Finally it appeared 
that the experience of participating in ETP training did lead to further investments in training. 
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Consortia and Training Agency Projects 



A specific purpose of our project was to look at the different types of ETP projects, including 
stand-alone employer contracts, consortia projects (where several companies work together 
to train workers), training agency projects teaching industry specific skills and training 
agency projects teaching generic skills. Our model showed that what distinguishes these 
different types of projects was their ability to customize training to each employer’s needs 
and to work closely with management to ensure that training was reinforced. We concluded 
that stand-alone employer projects had the greatest potential for improving productivity. 
Consortia projects also had high potential. Training agency projects, which taught industry 
specific skills to trainees from multiple employers, had somewhat limited potential for 
improving productivity. Training agency projects, which taught generic skills to trainees 
from a wide variety of employers and industries, had even more limited potential for 
improving productivity. 
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I Introduction 



Background 

The combination of an aging workforce, expanding international competition, and continuing 
technological change suggests that the need to train incumbent workers will increase. Today, 
45 states have some type of publicly funded incumbent worker training program. Two recent 
nationwide studies of state run incumbent worker training programs, one from the National 
Governor’s Association and one from the U.S. Department of Labor, recognize that ETP is 
the largest and most sophisticated program in the nation. Both reports also note that serious 
evaluation studies of these programs are sorely lacking. ETP is the exception. It has 
subjected itself to periodic independent evaluation of its impact on both individual trainees 
and the state s economy as a whole repeatedly over its history (Duscha and Graves, 1999; 
Regional Technology Strategies, 1999). ETP remains the nation’s largest state-funded 
incumbent worker training program and one of the most experienced. Its size and unique 
“pay for performance” policy makes ETP an important model for both public policy makers 
and business leaders around the nation. The purpose of this project is to better understand 
not only whether ETP has an impact on workers’ earnings and company productivity, but 
how ETP makes the impact it does. 

This project is shaped in part by our understanding of the pattern of training in American 
business. The most recent nationwide survey of employer-provided formal training shows 
that in 1993 only 71% of all employers provided any formal training for their employees 
(Frazis, Herz, & Horrigan, 1995). The use of formal training varied systematically by 
employer size. Ninety-nine percent of large employers (over 250 employees) provided 
formal training, but only 69% of small employers, (under 50 employees) did so. The pattern 
varied by industry as well. Employers in the finance, insurance and real estate sector were 
the most likely to provide training, followed by service industries, and transportation, 
communications industries, and public utilities. Manufacturing, the target of most ETP 
training, was less likely than average to provide formal training for employees, as were the 
construction and retail industries. The most common types of training, provided by 48% of 
employers were “job skills,” (technical skills directly related to a workers job), “workplace 
skills” (more general skills like total quality management techniques or just-in-time 
production techniques) were provided by 36% of employers. 

Training is not evenly distributed among workers either. Veum (1995) using Department of 
Labor data found that union members, better-educated, and longer-tenured employees were 
far more likely to be trained than other workers were. This follows a long stream of research 
clearly showing that employers invest the bulk of their training funds in college-educated, 
professional, and technical workers. For example, a recent comprehensive survey of 
employer-provided training by the Bureau of Labor Statistics found that only 60% of workers 
with a high school education or less received formal training from their employer in the last 
year, compared to 90% of employees with a bachelors degree or higher levels of education. 
The study also found the amount of training varied substantially. On average those with a 
high school degree or less received 10.9 hours of formal training compared to 16.1 hours for 
those with a bachelor’s degree or higher (Frazis, et. al., 1998). 
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Employer-provided training benefits both employer and employee. Since 1980 many studies, 
including several of ETP trainees, have found that employer-provided training boosts 
employee productivity, and increases worker earnings 5% to 12% annually (see Lillard and 
Tan, 1986; Hollenbeck & Wilkie, 1985; and Bartel, 1991; Moore, Blake & Phillips, 1994). 

These studies lead to a farther-reaching question: are employers providing sufficient training 
to keep American industry competitive in the long run? After an extensive international 
study, the federal Office of Technology Assessment (1990) concluded, “When measured by 
international standards, American workers are not well trained.” Many critics agree with this 
conclusion but debate why rational businesses would pass up investments in training which 
yield significant positive returns. Researchers speculate that employers fear their trained 
employees will be “poached” by other employers, or that many employers are not aware of 
the value of training. Perhaps employers would rather poach what they need than use the 
scatter shot approach of training a lot of workers who may not remain with the company. 

A recent study suggests that California employers invest less in training than similar 
employers do nationally. In a statewide survey of a representative sample of California 
employers with over 20 employers, researchers at the University of California, Los Angeles 
found that California employers were even less likely then employers nationally to invest in 
training. In all six categories of training studied; California lagged behind the nation for 
companies with over 250 employees. For example, nationally 58% of companies with over 
250 employees said they provided training in “production skills,” but only 32% of similar 
California employers said they did the same. The results for companies with between 50 and 
249 employees were more mixed, but only in the area of computer skills training was 
California consistently ahead of the nation (Erickson et. al., 1998). 

A final perspective that influenced our approach to the project is the growing understanding 
of the limited value of research and evaluation that looks narrowly at the impact of training 
on earnings. A comprehensive review of the research on public training programs concluded 
that: 

...information provided by current training program 
evaluations is quite limited. Nearly all training program 
evaluations are “black boxes,” indicating only whether a 
particular program “works,” on average, for a particular sample 
under a particular set of circumstances (including labor market 
conditions and service delivery systems). Such information, 
although useful, may not be readily generalizable to other 
programs, circumstances, or populations. 

This study attempts to open up ETP’s “black box” by looking in-depth at a sample of 
companies served by ETP to see how the training program interacted with the companies’ 
culture to shape the impact of training. From this new more sophisticated understanding of 
how ETP works, we generate recommendations for improving ETP’s performance. 
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Approach 



This study is far more ambitious than earlier studies of ETP’s impact in three ways. First, it 
makes a comprehensive evaluation of ETP’s impact on both the individual trainee and the 
company. Previous research has focused exclusively on direct impacts on individuals 
(Moore, Blake & Phillips, 1995). Second, by employing qualitative and quantitative 
methods, it attempts to uncover the dynamics of successful and unsuccessful projects and to 
explain the nature of ETP’s impact on the companies it serves. Finally, it takes a careful look 
at the dynamics of training agency and consortia projects that make up a growing share of 
ETP projects. 

This evaluation focuses on four aspects of ETP training: 

1 . The Quality of Training: This analysis looks at the quality of training 
delivered under ETP contracts including the quality of instructors, 
training materials, and the customization of training. 

2. Learning from training: This analysis examines the degree to which 
trainees mastered the material taught in training and the degree to 
which they were able to use it on the job. 

3. Reinforcement of learning from training on the job: This analysis 
looks at how effectively learning from training was reinforced once 
trainees returned to their job. 

4. Impact of training on companies and individuals: This analysis 
measures both qualitatively and quantitatively the impact of ETP 
training on both the individuals trained and the companies served by 
ETP. 

Overview 

First, we summarize our research methods. Next, we present the results of our evaluation. 

The results are divided into two major sections. The first section presents a model describing 
the factors that influence the impact of ETP training. Then using the results of our fieldwork, 
we show how these factors interact in specific projects to determine the success or failure of 
individual projects. 
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II Methods 



Qualitative Methods 

We chose a case study approach for this part of the study because we wanted to examine 
some ETP contracts in-depth and capture the multi-dimensional dynamics of the projects. 

We wanted to gain insights about how ETP projects work to better interpret the data from our 
large-scale analyses of earnings and company finances. The cases were selected to represent 
the variety of the types of projects that ended during the 1995-96 fiscal year. They include 
training agencies, consortia, and individual (stand-alone) company contracts. The field study 
methods allowed us to make qualitative observations of the companies in question and to 
examine each company’s training experience. 

A final note: in order to protect the confidentiality of the companies we visited we changed 
their names and at times their locations. All data reported, however, are completely factual. 

• Sample 

We chose a purposeful sample, which was designed to capture the wide variation in ETP 
projects that train incumbent workers. Purposeful samples are valuable in evaluation studies 
because they allow the researcher to capture the variety of types of cases while using a 
relatively small number of cases. Evaluation experts recognize purposeful samples as a valid 
method for describing both excellent and problematic programs without attempting to 
generalize to an entire population (Patton, 1980). 

We selected a purposeful non-random sample of companies with the intention of balancing 
the sample along three key variables: 1) whether the project was a consortium, training 
agency, or a stand-alone project; 2) the reason for training; and, 3) the size of the company 
served. In selecting particular companies we also considered geography (e.g. Northern 
versus Southern California), contract size, and whether the company was in the service or 
manufacturing sector. In the case of training agency and consortia projects, we next selected 
three companies served by the project for a full field visit. A summary of companies 
included by sampling criteria is presented in Table 1 . 
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■■ ;-v : ^ Table T Y-S: ; 
Fieldwork Sample ' 




Stand Alone Projects By Reason and Employer Size 


Number of 
Companies 


Preventing Displacement 




< 50 Employees 


0‘ 


50-100 Employees 


1 


101-250 Employees 


0 


250+ Employees 


3 


High Performance Workplace 




< 50 Employees 


1 


50-100 Employees 


1 


101-250 Employees 


0 


250+ Employees 


2 


Consortia Projects 




Training Agency Consortia 
(4 projects) 


12 


Business Consortia** 
(2 projects) 


3 


Total Companies In Sample 


23 



* There were no projects in this category during the sample year. 

**One contract that was listed a business consortium in fact served only one business; hence 
only one company was visited under that contract. 



• Field Methods 

After selecting the sample companies, ETP sent each a letter introducing our research team 
and project. We contacted a representative from each contract to set up interview 
appointments. Each company was informed of the purpose of our study, the activities we 
would conduct, and the amount of time needed for the visit. We provided our own Spanish 
translator when necessary, and translated all evaluation questionnaires into Spanish. 

S Interviews 

We met with the managers who developed the ETP contract to discuss their motivation for 
undertaking training, the extent they felt their objectives were met, and their view of the short 
and long-term impact of the training on the company. We also asked about their relationship 
with ETP staff. In addition, if there was a contract trainer or consultant on the project, we 
attempted to interview them. 

S Focus Groups 



When possible, we conducted at least two focus groups consisting of four to nine trainees, 
and individual interviews with two or three supervisors of these trainees. The purpose of the 

t best copy available 
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focus groups was to capture the experience of trainees, both during training and since 
training. 

^ Evaluation Questionnaires 

According to Donald Kirkpatrick’s Evaluating Training Programs: The Four Levels, training 
may be evaluated at each of four levels: 

1 . Evaluating Reaction : The evaluator simply asks the trainees for the perceptions of 
the quality and value of the training typically through a structured questionnaire 
immediately after the training. 

2. Evaluating Learning : The evaluator assesses whether the trainee mastered the 
skills, knowledge, or behavior the training was meant to impart. This can be 
measured by observing trainees on the job or collecting supervisors’ rating of 
trainee performance. 

3. Evaluating Behavior. The evaluator seeks to measure what portion of the skills, 
knowledge or attitudes the training intended to teach that are actually used on the 
job. 

4. Evaluating Results: At the highest order, evaluators look to see whether the 
training had an impact on the trainees’ performance at work. This can be 
measured in a variety of ways specific to the company: scrap rates, reduced 
conflict on the job, reduced absenteeism, increased productivity, etc. This method 
can go as far as estimating a Return on Investment (ROI) for training. 

In conjunction with the interviews, we administered two separate questionnaires, one to 
trainees and a second to their supervisors. Both questionnaires were designed to gather data 
regarding all four levels of evaluation. The trainees evaluated training quality and 
commented on their resultant changes in productivity, if any. We also asked whether training 
led to changes in the work environment. The supervisors completed a similar questionnaire 
regarding their opinion of their trainees’ learning effectiveness and change in productivity. 
Each company’s survey results were tabulated and given back to the company so they could 
benefit from our fieldwork. Sample questionnaires are included in Appendix A. 

S Observations 

We walked through the manufacturing process to observe trainees at work and to note any 
effects of training. For example, we often were able to see SPC charting at work, team 
meeting areas, team notes, or the use of new technology on which trainees were trained. 

S Document Analysis 

ETP provided copies of the contract files for all clients in the sample study. We focused 
specifically on the number of trainees retained from the contract period, the type of training 
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that occurred, and the industry represented by each. We also read the field reports of project 
monitors to get an understanding of issues that emerged during training. In addition, we 
examined any available curriculum materials, or other materials related to training such as 
SOST projects, quality team minutes, etc. 

• Limits 

S General Caveats 

As mentioned earlier, our research sample is purposeful and not designed to be generalized to 
all ETP projects. Similarly the data generated by the evaluation questionnaire are based on 
the population of trainees who remained with the companies until we conducted the 
fieldwork. They may not be representative of all the trainees in that particular project or 
other ETP projects. 

S Cooperation Problem 

While the majority of companies selected for the sample were cooperative and helpful in 
allowing us to visit them, unfortunately quite a few companies declined to participate. Two 
companies in the “preventing displacement” category declined to participate. The most 
problematic was the high-performance workplace category. The company selected in the 
101-250 enrolled size category was replaced four times. Another company in the 250+ 
category had to be replaced twice. While companies in their ETP contract agree to cooperate 
with follow-up research, our university code of ethics forbids us from using any form of 
coercion to get cooperation in research projects. 

S Incomplete Performance Data 

This is related to cooperation problems. Once we were on-site, we found it impossible to get 
all the data we desired. This problem had two basic causes: First, most of the companies 
were private and unwilling to share detailed financial and productivity information. 
Fortunately, they often shared enough partial information that we were able to make 
estimates of the impact. Second, the companies did not attempt to measure the impact of 
training and therefore had nothing to give us. Our own detective work allowed us to ferret 
out some information. This work helped alleviate the performance data problem to a certain 
degree. A complete discussion of how we assessed the impact of ETP on the companies we 
studied is included with the discussion of ETP impact in Section V. 
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Ill An ETP Training Model 



The Model 

Qualitative fieldwork allowed us to capture the complexity of the ETP training projects we 
studied. The results of our extensive analysis of the 23 contracts studied can be summarized 
in a model (displayed below) that captures conceptually the basic dynamics of how ETP 
training programs do or do not lead to increased company performance. 



Figure 1: Training Impact Model 
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There are two key points about the model: it is sequential and multiplicative. It is sequential 
in that each element appears in the chronological order in which they occur in a training 
program. Before training, a company’s production has some existing relationship to the 
industry’s best practice. Obviously, the planning of training precedes the actual training. 
Next, the reinforcement of training must come after the training. Finally, “Potential Gains” 
can be conceptualized as value of improved productivity in dollars; the “Quality of Training” 
and “Management Reinforcement” can be measured on a scale of 0 to 1 , or in percentages. 
Thus if both the “Quality of Training” and “Management Reinforcement” received are 1, 
100% of the “Potential Gains” would be realized. 

The model is multiplicative in that the quality of each component amplifies what has come 
before. For example, relatively poor training with strong reinforcement of what was learned 
will still have a significant impact. Conversely if any of the factors are completely absent, a 
value of zero, there will be no impact. For example, if there is absolutely no quality in the 
training, value 0, there will be no impact regardless of how well planned or how much 
management reinforced the training. 

Here is a complete explanation of the model. 

• Potential Gains 

A company’s potential gains from training are limited by the difference between the 
company’s current practices and the industry’s best practices (Industry’s Best Practices - 
Company’s Current Practice). For example, if the application of new quality techniques 
produces processes within the industry with scrap rates of only 1 %, and a given company has 
a scrap rate of 10%, costing them $1,000,0000 annually, then their potential gain from 
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coming up to the industry’s best practice is $900,000, a 90% reduction in scrap. 
Paradoxically, a company near the industry’s best practice, say a 2% scrap rate, will only 
have a scrap cost of $200,000, and thus gain only $100,000 by coming up to the industry’s 
best practice, still a 50% reduction. While it is possible for companies to achieve even more 
dramatic results through innovations that improve on the industry’s best practice, our 
fieldwork shows that, most commonly, companies served by ETP are striving to come up to 
the best practice level rather than make technological breakthroughs. 

• Quality of Training 

The quality of training has two components, the training design and the training delivered. 
We discuss each component separately. Our model implies that they are multiplicative. This 
means that if the training actually delivered has very low quality, it will lead to very low 
quality training overall, even with good planning. The converse is also the same: a poorly 
planned training program well-executed will lead to poor quality training overall. 



S Design 

Decisions made in the design of training have an important influence on the ultimate impact 
of training. We learned from our fieldwork that key design elements include: 



• selecting workers for training, 

• selecting skills to be taught, 

• selecting level at which skills will be taught, 

• timing training in relation to other changes such as introduction of technology, 
quality teams, or incentive pay, 

• plans to institutionalize training, 

• the decision to use in-house or contract trainers or both, and 

• how the amount of planning invested reinforces changed behavior after the 
training. 



All these factors will be discussed later. For now, one example illustrates the importance of 
planning the level of training. We found several cases where SPC training was too advanced 
and theoretical, requiring math skills many trainees did not possess. The trainees became 
discouraged and increasingly negative about training in general, because of this experience. 
Thus, in this case we found that the training had little ultimate impact on the company’s 
performance because the trainees were unable to master the skills. In other cases where 
trainees were taught basic SPC skills after receiving a math refresher course, they were able 
to immediately put their knowledge to use in production. Here, the implementation of SPC 
had a significant positive impact on productivity. 
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S Training Delivery 



Good training delivery also matters. Our results show that high quality ETP training has 
several key characteristics: 1 ) It targets an appropriate level for the trainees; 2) it is 
customized to the company; and 3) it has effective instructors. In addition, good quality ETP 
training is able to communicate intangible messages to workers such as “the company cares 
about the worker,” “the worker has an opportunity to advance and improve,” and “the 
company as a whole is moving forward.” The ability of training to carry these intangible 
messages seemed particularly important in companies that had suffered a series of 
downsizing, and were now moving forward. 

This measure also assumes that where there is good training, there is learning. We therefore 
conceptualize this variable as synonymous with learning. In fact a measure of training 
quality can be the amount of useful skills and knowledge that trainees learned. 

One thing we observed about this variable is that while the quality of training varied 
substantially, we found only one ETP training program whose quality was so poor that it 
actually undermined productivity. Otherwise, all training seemed to hold at least some 
potential to improve productivity. 

• Management Reinforcement of Training 

Many studies have found that management involvement is a key to effective training. 
Through our fieldwork, we gained additional insights into the critical role that management 
reinforcement of training played in ETP contracts. We see management reinforcement 
occurring along several dimensions: 



^ Messages about the value of training: If management sent clear messages to 
supervisors and workers that training was important and valuable, then the 
training was much more likely to have a significant impact. Oftentimes, the 
message was more than just words. For example, we found a company president 
who taught the basic math course for SPC himself, sending a powerful message 
about the degree to which he valued training. 

^ Reinforcement of skill use and techniques: Trainees must have a chance to use 
skills on the job, otherwise no amount of learning will make a difference in 
productivity. If employees return from training to find that the new technology 
they were trained to use is not in place, the training will have no impact. 
Similarly, training in “soft skills,” such as decision-making or other TQM 
techniques will have no impact unless teams are formed, assigned problems, and 
given an opportunity to meet soon after training. 

S Timeliness of Reinforcement: We found training has a dramatically short shelf 
life. Even in cases where management signals that training is important, and is 
willing to invest in new technology or reinforcing skills in other ways, if the 



O 

EKIC 



ETP At Work 



19 



10 



reinforcement does not come promptly, the impact of training will diminish 
dramatically. For example, in one factory we found two groups of SPC 
trainees — one that used their SPC skills effectively and one that did not. Both 
groups had identical SPC training. The group that used the skill effectively 
immediately implemented SPC when they returned to the production line after 
training. The group that did not use SPC effectively had to wait several months 
for SPC to be implemented in their production process, and by that time they had 
lost their mastery of the skills. 



How the Model Works 

The workings of the model are illustrated in three actual cases from our fieldwork. The 
examples include a very successful project, a partially successful project, and an unsuccessful 
project. First, we assumed a particular potential gain from training for illustrative purposes 
only. Next, based on our field observations and survey data, we estimated a percentage value 
for each of the variables in the model to indicate its performance in the particular project. 
Then we calculated the theoretical impact of training based on the dollar value of the 
potential gains realized. 

S Very successful project : This company manufactured pumps. Its productivity was 
closer to the industry’s best practices than most companies, but it still had 
significant room for improvement. Management worked hard to design 
customized, high quality training, and they delivered it well. The TQM, SPC, and 
other training were followed by a carefully planned implementation of TQM 
teams and SPC practices on key production lines. Scrap rates and warranty work 
dropped dramatically. The following model illustrates the impact of this project. 



Potential Gains 

(Best practice - 
current practice) 


X 


Quality of Training 

(Training Design x 
Quality of Training 
Delivered) 


X 


Management 
Reinforcement 
of Training 


= 


Value of 
Potential 
Gains 
Realized 


$400,000 


X 


90% 


X 


90% 


= 


$324,000 



V Partially Successful Project: This company made components for heavy 

equipment. It was far below the industry standard. It hired a consultant to deliver 
all of the training and for various reasons the quality of the SPC and TQM 
training was poor. After training, management did follow-up, creating quality 
teams and allocating resources to make innovations recommended by the teams. 
As the table shows, despite the poor quality of training, the large potential gains 
and good management reinforcement led to significant productivity gains. 
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Potential Gains 
(Best practice ~ 
current practice) 


X 


Quality of Training 
(Training Design x 
Quality of Training 
Delivered) 


X 


Management 
Reinforcement 
of Training 


= 


Value of 
Potential 
Gains 
Realized 


$900,000 


X 


20% 


X 


90% 


= 


$162,000 



■S Unsuccessful project : The unit in which training took place processed claims and 
provided customer service for a large insurance company. Through a 
combination of in-house and contract trainers, the company carefully planned a 
large scale training intervention and then provided excellent quality training to 
almost all of its employees. Immediately after training, employees were 
encouraged to tackle difficult quality problems and were provided resources. 
However, after a short period the unit was reorganized. Most trainees were 
relocated, and new management stopped reinforcing the practices. 



Potential Gains 
(Best practice - 
current practice) 


X 


Quality of Training 
(Training Design x 
Quality of Training 
Delivered) 


X 


Management 
Reinforcement 
of Training 


= 


Value of 
Potential 
Gains 
Realized 


$900,000 


X 


90% 


X 


0% 


= 


$0 



Lessons from the Model 

The model quickly illustrates a few lessons from the fieldwork. First, the companies’ 
potential gains from training vary substantially. There are limits on the degree to which 
companies can actually improve their productivity. Companies far down the proverbial 
learning curve actually have more potential to benefit from training than better-run 
companies which have already achieved a substantial portion of the potential productivity 
gains. Second, high quality training contributes to the potential gains from training but does 
not guarantee productivity increases. Finally, management behavior controls the degree to 
which potential gains are actually achieved. As the model suggests if there is no 
management reinforcement (a value of 0), there will be no change in productivity. 
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IV Field Results 



In this section we use the training model to analyze our field results from the 23 Companies 
we visited. The organization of this section follows the sequence of the components in the 
model, so we address the potential gains from training first. 



Potential Gains from Training 



Potential Gains 
(Best practice — 
current practice) 





Quality of Training 




Management 




Value of 


X 


(Training Design x 


X 


Reinforcement 


_ 


Potential 




Quality of Training 




of Training 




Gains 




Delivered) 






Realized 



In our field visits we were struck by the wide variation in the potential of companies to gain 
from ETP training. The basic paradox is this: companies with strong management are likely 
to have achieved a large proportion of the potential gains from training. These well-run 
companies have already improved their production process and thus have less potential for 
benefiting from ETP training, yet these same companies may be best able to obtain and 
execute an ETP project because they do have strong management. Conversely, companies 
far below the best practice standard for their industry, due to weak management or other 
problems, have the most to gain; but they are likely to have the most problems executing an 
ETP funded program and supporting changes in the production system. 

A related observation is that companies that are at the best practice level will need break- 
throughs to improve productivity. These are less likely to come from training than from 
technological or systems changes. Also, training investments that are intended to 
breakthrough current industry practices are high risk since the ideas are new and untested. 
They do not have a proven track record, and thus they may not yield any results. 

What we observed in the field is that companies with relatively low quality and inefficient 
production processes experienced tremendous gains from basic training in techniques like 
TQM, basic production planning, or decision making. Implementing even rudimentary team 
quality practices led to substantial improvements in quality and productivity. In other 
companies that were much better managed, with better-trained workers, substantial training 
in relatively higher order skills, and careful implementation of management level reforms led 
to significant but smaller improvements in quality and productivity. See the text box below. 
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A Tale of Two Companies 

Located just miles apart in a central valley city are two companies each with ETP 
contracts which represent the wide range of potential gains from training. 

The first company, T-Bar, manufactures roll bars for heavy equipment and 
related products. Production is carried out in an open air shed, which appears 
dirty and disorganized. The company has grown rapidly. Prior to ETP training 
there was little focus on improving the efficiency of production. Workers simply 
followed traditional methods for production and pushed the product out the door. 
For example, assemblers kept parts in a helter-skelter array of boxes on the floor 
around the assembly area. The process was obviously far below the industry best 
practices for any manufacturing process. After basic TQM training, management 
formed teams and allocated resources based on suggestions from the teams. The 
team of assemblers organized their parts on shelves in a container near their work 
area. The simple change dramatically improved their efficiency and ability to 
keep track of parts. Thus T-Bar experienced significant improvements from this 
and other basic changes because they were far below the industry standards. 

A few miles across town, Flow Pumps produces stainless steel pumps in a clean, 
modem, air-conditioned factory. Management has invested in state-of-the-art 
equipment. The highly trained human resources staff carefully selects employees 
for motivation and basic skills. Production is organized by an in-house staff of 
industrial engineers to maximize efficiency and quality. Still the company 
wanted to improve, so through ETP it invested in training its frontline production 
workers in SPC and TQM techniques. After training it had teams attack difficult 
lingering problems many of which were eventually solved, leading to important 
but relatively small gains in productivity. The gains were relatively small 
because the production processes were already running at close to the optimum 
level. 
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Quality of Training 



Potential Gains 

(Best practice — 


Quality of Training 
X (Training Design x 


X 


Management 

Reinforcement 




Value of 
Potential 


current practice) 


Quality of Training 
Delivered) 




of Training 




Gains 

Realized 



Our model as derived from the fieldwork, revealed that the quality of training had two basic 
elements: the quality of planning that went into the training and quality of the training 
actually delivered. In this section we will discuss the elements that make up effective 
planning first and then examine the quality of training delivered by the projects we studied. 

• Planning Training 

From the fieldwork we identified a series of key activities and management decisions that 
have a critical influence on the quality of training. These steps are listed below with our 
observations about each one. 

S Clear Objectives 

As with any major undertaking ETP training projects need clear goals to be effective. In our 
fieldwork we confirmed that companies which had clear explicit goals were more likely to 
complete their ETP contract in a timely and effective manner. In some cases external forces 
drove the goals. For example, we saw several cases where companies had to train to achieve 
quality certification from a third party, such as Boeing or some other major contractor. In 
these cases we found the projects were clearly focused and had top management’s attention, 
although they sometimes lacked the customization needed to be optimally effective. In cases 
where goals were either unstated or vague, for example, “we want to train to improve 
quality,” we found that when conflicts occurred between training and other goals (such as 
production), training suffered. 

Similarly, if the goals were not understood or shared by all managers, then often conflict 
erupted when the resource requirements of training became clear. We observed that if 
trainees did not understand the overarching goal that drove training they were less motivated 
and were unable to put training in context. Clear goals also help companies make other 
essential training decisions such as which training topics to cover, and whether or not to use 
in-house trainers. 

S Assessing basic skills 

Assessing the current skill levels of trainees is a key factor in planning appropriate training, 
but this step is commonly overlooked in the projects we visited. For example, we found 
repeated instances where companies attempted to teach SPC techniques to frontline workers 
who lacked the basic math skills to grasp or use the techniques. Companies believe that they 
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know their workers well, but we found they often misjudged employees’ skills because they 
did not use systematic assessment of individual workers. Some consultants use systematic 
needs assessment, others do not. Sometimes the experience in training leads to improved 
skills assessment. For example, after having difficulty teaching SPC to its existing 
workforce one company began screening new hires for basic math skills and added a course 
in basic math skills to the training all new hires receive. 

^ Consultant versus In-house Training 

A major strategic decision made in the design of training is whether to hire training 
consultants or use in-house trainers. The issues surrounding the use of consultants and 
contract trainers are complex and explored in an earlier study, (Moore, Blake, et. al., 1997). 
In our fieldwork we saw many consultants and contract trainers with industry-specific 
expertise, who knew ETP well, and delivered high quality programs. There were trainers 
who took the time to familiarize themselves with the company’s processes and then 
customized instruction to the company. Unfortunately, we also found trainers who poorly 
delivered generic training to trainees who were not well prepared. 

The earlier study of consultants and contractors showed that most employers do not shop 
effectively for consultants and training contractors. ETP’s continued efforts to encourage 
employers to carefully evaluate consultants and contract trainers should eventually improve 
the planning of projects. 

Our most telling observations were about the value of choosing to do training in-house. 
Companies that either chose to do the training in-house or teamed employees with outside 
trainers received many benefits. First, the inside trainer’s intimate knowledge of the 
company appeared to improve the quality of training. Next, by using senior people as 
trainers, the company sent a powerful message that training was valued and important to 
upper management. In cases where frontline workers were teamed with managers to serve as 
trainers, the cooperation between workers and management in training modeled the new 
cooperative environment the training was to produce, again sending an important message. 

Finally, we observed that training is much more likely to persist beyond the ETP contract in 
cases where training was provided by in-house trainers. There are some simple reasons for 
this. With in-house training there is more likely to be a customized curriculum owned by the 
company. Trainers are there and while they must take time away from regular duties to train, 
it is less expensive than hiring outside contractors. Using in-house trainers also means that 
the same trainers will teach multiple cohorts of employees over time, helping to infuse 
training into the culture of the company. In general, using in-house trainers seems to 
increase the company’s commitment to training. 

•S Intact versus Mixed Training Classes 

A key planning decision is whether to train intact groups of workers or mixed groups of 
workers. In our fieldwork we found that successful projects had a conscious strategy of 
forming groups which would reinforce larger project goals. For example, one company 
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whose core process was processing insurance claims knew it had a communications problem 
between units, which was undermining quality. The company deliberately created training 
classes with employees from multiple departments. The trainees built relationships with 
people from other departments in class and learned about how they saw problems and 
barriers. Back on the job, these new relationships and perspectives led to a major 
improvement in communications. Conversely, in another project the focus was on creating 
quality teams to solve problems within various production areas. Classes were made up of 
entire work units. In the class, teams were formed and they began to work production 
problems in class. When training ended, the teams were already formed and functioning, 
enabling them to immediately begin to solve problems and improve quality. 

S Planning Reinforcement 

We discuss the importance of management strategies for reinforcing training at length later in 
the report. Here, we just want to note that we were struck in the fieldwork by how few 
companies consciously plan to reinforce what was learned in training. It is well known that 
changes in behavior promoted by training will not persist unless they are reinforced after the 
training is over. A simple example is that if training teaches a variety of group problem- 
solving techniques and quality strategies, but groups are not given the authority to take on 
problems, the time to meet, and the resources required, then training will yield few quality 
improvements. 

Most companies we visited that did eventually develop effective reinforcement strategies did 
so in an ad hoc manner after the training was over and managers realized they needed to do 
something more to make training pay off. In the few cases where systematic reinforcement 
was planned, we found the effects of training to be potent and the payoff to training 
immediate. 

S Importance of Creating Just-In-Time Training 

The fieldwork impressed on us the importance of training to be delivered “just in time.” Any 
experienced teacher knows that the shelf life of new learning is short. If new skills or 
knowledge are not used promptly they are lost no matter how good the initial training. We 
saw many examples in our fieldwork where companies provided good training under ETP, 
but when workers returned to the production line, the systems in which they would use these 
skills were not in place. For example, when a group of production workers in one company 
were trained in SPC, there were no gauges or charts for them to use to implement SPC when 
they returned to the production lines. Only after six months was an SPC plan ready for their 
part of the production line; by that time, most trainees had forgotten the SPC they had 
learned and they had a great deal of difficulty getting the SPC system going. Much of the 
benefit of the original training was lost. Similarly, training workers on new technology long 
before the technology is in place is futile. 
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'S Planning SOST to expand and reinforce training 

We will discuss SOST 1 at length later in this report. At this juncture we would like to note 
that the planning of SOST was much more problematic than the planning of classroom 
training. Many projects included substantial SOST hours, but the planning required to ensure 
the quality of SOST was often absent. Often trainers had not designed SOST assignments 
specific to the production process, or provided coaching while trainees worked on the 
assignments. Finally, supervisor commitment to allow employees time away from 
production to complete SOST was often missing. 



PH A Insurance: Good Management Planning Leads to Effective Training 

In 1994 the managers at PHA Insurance (not company’s real name) wanted to 
become NCQA certified (this is like an ISO 9000 quality certification). They 
went through the certification process and did not pass. “It was a big blow to the 
company,” according to the managers interviewed. 

As a result of this failure, managers “wanted to improve the staffs ability to 
make decisions and make change.” According to the managers, training had 
three goals: 

1 . Make a visible improvement in the business process. 

2. Achieve a “clear” NCQA accreditation in three years. 

3. Improve customer service. 

The managers also recognized that to be effective in the long run, they needed to 
increase communication across the various departments and institutionalize the 
training. “We wanted to establish relationships we wouldn’t normally have.” 
“Training was to create an opportunity for dialog; people discovered who was a 
customer and who was a supplier.” PHA submitted a proposal to enroll 600 
trainees and complete training for 500. Training would be in three areas: 
Management Skills, SPC, and Office Automation. 

Management selected a widely recognized industry organization to provide a 
tailored curriculum for the project. The organization offered a wide array of 
training modules. Based on the manager’s experience with the failed attempt to 
achieve quality certification, they selected modules that would eliminate the 
organization’s deficiencies. The modules were then tailored to PHA’s particular 
processes. The organization also trained existing staff to serve as trainers for the 
modules selected. This decision was made to insure that after the ETP training, 
training for new hires and refresher training would be available. The managers 
also believed that trainers who know the organization’s culture would be better 
able to reshape it through training. Within six months of the training the 
organization achieved NCQA certification. 
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Not all projects are successful. The following text box illustrates how poor planning led to 
an unsuccessful project. 



Wavelength: When Badly Planned Training Gets Worse with Management 
Apathy 

Wavelength manufactures precision radio components and subsystems for the 
military, intelligence, and commercial sectors. As the defense industry 
contracted, Wavelength was forced to move into the commercial 
communications market, where the profit margins are lower and customer 
demands for quality, service, and response time much higher. In response, the 
management set out to establish a “high performance workplace.” They decided 
to train their 200+ workforce in Vocational English-as-a-Second Language, 
Statistical Process Control, Management Skills, and Manufacturing Resource 
Planning. The result, they hoped, would be for front-line employees to be 
empowered, and managers to be equipped to adapt to those changes. 

The managers and supervisors filled out questionnaires regarding what they felt 
was needed in terms of training, and a senior human resources manager selected 
a contractor. Trouble began when the trainees lost interest during classes. Part 
of the problem was that the curriculum was “over their heads.” Also, it was 
unclear to the managers how much training would disrupt their work. The 
resulting production delays were further exacerbated by SOST, which was 
perceived as having little value and was required to be completed during work 
hours. 

The situation did not become catastrophic, however, until Management lost 
interest. Instead of working with the instructors to adjust the curriculum, the 
managers just began leaving in the middle of class or stopped attending 
altogether. Employee morale began to plummet. TQM became the least useful 
component to training simply because the managers never gave the workers an 
opportunity to practice it. 

On the questionnaires, trainees expressed strong positive responses to the 
opportunity, to learn new skills and increase their motivation. Thirty-four percent 
agreed and 34% strongly agreed to the statement that they “feel more motivated 
and involved” due to the training. 

However, clearly their worst memories of the training included the bad match 
between what was taught and what they could apply to their work, and 
abandonment by their apathetic management. 



SOST stands for Structured On Site Training. This is the component of training where trainees practice skills 
learned in class on the job under supervision. It is a component of most ETP funded training. 
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• Qu ality of T raining Delivered 

We begin this section by summarizing data from our evaluation survey, which was 
administered to trainees at every project we visited to provide an overview of the quality of 
ETP training. Next, using both the survey data and the fieldwork we examine three critical 
quality issues that emerged during our delivery of training delivery: 

■S Effectiveness of trainers 
S Customization of training 
S Effectiveness of SOST 

As noted earlier, as part of the fieldwork we asked trainees to rate the quality of the ETP 
training they received on a standard evaluation questionnaire. The results show that trainees 
rated the overall quality of training as slightly better than “good,” giving it a rating of 3. 1 1 on 
a 4 point scale (see Figure 1). The trainees rated nine aspects of training. The highest rated 
aspects were the quality of instructors and their ability to hold the trainee’s interest, along 
with the clarity of the training objectives and the quality of instructional materials. Lower 
rated aspects included the degree to which the training was customized to the company, the 
effectiveness of SOST (structured on-site training), the time dedicated to each topic, and the 
level of training. 

Results varied substantially from one company to another. For example, at some sites 
trainees were very pleased with the effectiveness of SOST, and very critical of the 
instructional material. Overall field interviews and observations confirmed that the quality of 
instructors tended to be good, while the effectiveness of SOST and the degree to which 
training was customized to the company differed substantially among the companies. 



Figure 1 

Trainee Rating of Training Quality 



Quality of Training Overall 
Time on Each Topic 
Effectiveness of SOST 
Degree Training Customized to Co. 

Right Level for Trainee 
Usefulness of Topics 
Ability of Trainers to Hold Interest 
Quality of Instructional Materials 
Clear Objectives 
Quality of Instructors 

1 



■ Average Rating 1=Poor, 2=Fair, 3=Good, 4=Excellent 
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To explore the relationship of the quality of training to the impact of training, we used 
regression analysis. Regression is a statistical technique that measures the strength of the 
relationship between predictor variables. As the table below shows, we used regression to 
see if different aspects of the quality of training were associated with the impact of training. 
The impact of training was measured three ways: the amount learned, how often new skills 
were used, and the impact of training on productivity. The coefficients measure the strength 
of the unique relationship between measure of quality and the outcome measure. The 
significance measures show the probability the measured relationship is due to chance or 
random error. Thus only levels of significance of 0.05, which indicate that there is a five- 
percent or smaller probability that the relationship is due to chance or random error, are 
considered statistically significant. Significant relationships are shown in bold type. Finally 
the overall R measure indicates the proportion of the variance in the impact measure 
accounted for by all the measures of quality. This measure shows the degree to which the 
measures of quality predict the impact of training. The higher the R 2 the stronger predictor 
quality measures are of the impact of training. 
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Table : 2 : ip.' 



Quality Measure 


Amount Learned 


How Often Skill 
Used 


Impact on Productivity 


Coefficient 


Significance 


Coefficient 


Significance 


Coefficient 


Significance 


Clear objectives 


.048 


.481 


.028 


.701 


.044 


.527 


Usefulness of topics 


.159 


.013 


.311 


.000 


.249 


.000 


Length of time on 
topics 


-.034 


.620 


-.059 


.409 


.041 


.566 


Quality of instructional 
materials 


.065 


.315 


.067 


.316 


.001 


.983 


Degree of 
customization to 
company 


.127 


.043 


-.060 


.356 


.086 


.166 


Quality of instructors 


-.145 


.056 


-.124 


.126 


-.028 


.712 


Effectiveness of SOST 


.306 


.000 


.344 


.000 


.258 


.000 


Ability to hold interest 


.164 


.032 


.039 


.624 


.082 


.283 


Right level 


-.037 


.554 


-.055 


.391 


-.089 


161 


Quality Overall 


.040 


.612 


.101 


.220 


.093 


.240 


Adjusted R z 


.284 




.310 




.343 





Overall the three regressions analyzed accounted for between 28% and 34% of the variance 
in the impact measures. All the models were statistically significant, indicating that the 
quality of training had a significant relationship with the impact of training, but there were 
many other unmeasured factors involved as well. Interestingly, the strongest relationship (R 2 
= .343) was between the quality of training and the impact on productivity, indicating that 
better quality training leads to larger increases in productivity. Relationships between quality 
and how often new skills were used (R 2 =.310) and quality of training and amount learned 
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(R- = -284) were significant but weaker. Overall these results support the training model we 
introduced earlier. Individual measures of quality which had strong relationship with all 
three measures of the impact of training were “usefulness of the topics” covered in training 
and the “effectiveness of SOST.” We discuss SOST at length later in the report, but it is 
important to note here that these results show that when the “effectiveness of SOST” is rated 
highly by the trainees they also report larger positive impacts of training. Conversely, when 
SOST rated poorly the impact of training was rated lower as well. The “degree of 
customization to the company” and “ability of the training to hold interest” were also 
significantly associated with amount trainees reported learning. 

S Trainee Learning 

A second key indicator is whether or not trainees learned what was taught in training. Again 
the results show that overall ETP training was successful; over two thirds of the trainees 
report that they were able to learn “most” or “everything” of what was taught. Less than two 
percent of trainees said they learned none of what was taught. 

Again the results varied substantially from company to company and from one skill area to 
another. For example, trainees trained in TQM report mastering a larger proportion of what 
was taught than trainees who took CAD/CAM (computer-aided design/manufacturing). 



Figure 2 

Am ou nt Trainees Learned 
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S Effectiveness of Trainers 



As anyone who has ever taken a class knows, the effectiveness of the instructors is a key to 
the class’ success. We have good news here in that the quality of instructors was the highest 
rated item on our survey (3.28 for instructors versus 3.1 1 for overall training). In our 
interviews we found many instructors (both contract and in-house) to be motivated, skilled, 
and highly committed. We found many instances where even though trainees criticized the 
training, they praised the instructors. When confronted with a generic curriculum or 
materials which were pitched to an inappropriate level, the skilled instructors often attempted 
to change the instruction on-the-fly to make it more effective. We found trainees responded 
well to instructors who were knowledgeable about their company’s industry and took an 
interest in the trainees as individuals. The individual sensitivity was particularly important to 
trainees who had limited formal education and who were very anxious about being in a 
classroom situation. 

S Customization 

The fieldwork convinced us that the customization of training was critical to creating 
effective ETP training. Customization needs to occur on three levels. First, as we noted in 
the planning training section, the level at which topics is taught needs to be adjusted to the 
level of trainees. Teaching college-level SPC techniques to trainees with limited math skills 
is futile. Second, the training needs to be customized to the unique processes used by the 
company. Examples pulled directly from the trainees’ daily experience are much more 
powerful teaching tools than generic examples. Finally, training needs to be in tune with the 
company’s culture. If a company operates as a rigid hierarchy with a great social distance 
between workers and managers, training “which models open informal communication” will 
not be successful since these approaches will not be used on the job. 
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Trophies Galore: Customized Training = High Quality Training 

Trophies Galore is a manufacturer of plastic components for trophies. In order 
for this family-owned business to remain successful, it had to maintain its 
product innovations and improve its service. 

Trophies Galore’s management knew it needed to upgrade the skills of all its 
employees to remain competitive. Management had considered hiring a training 
consultant to design a curriculum but after learning about a business association’s 
consortia program, they chose to join that instead. The association connected 
them with a consultant who was also a trainer. Before committing to that 
consultant, however, one of the company’s executives sat in on one of his classes 
at another site. Further, this executive reviewed the curriculum, and arranged for 
the trainer to tour the plant and learn the processes, so that an understanding of 
the company’s processes and culture could be built into the training. In short, the 
management wanted to make sure training would be customized to the 
company’s needs. 

Fifty percent of the trainees rated the quality of training as excellent. Fifty 
percent also rated the quality of instructional materials as excellent. “Jim (the 
instructor) added a great deal of interest. Even the exercises were like games.” 

Over 75% of the trainees rated the customization of training as “good” or 
“excellent.” “The instructor was able to customize to our company.” “He got to 
know us on an individual basis and he also learned the manufacturing process.” 

“It was a very well put together program, and equally important, the selection of 
the trainer was exceptional.” 



As previous results show, trainees rated the customization of training below their overall 
quality ranking (2.9 for customization versus 3.11 overall). The lack of customization in 
many ETP projects has its roots in several factors. First, managers often don’t understand the 
importance of customizing training nor how training can be customized. To simplify the 
process for themselves they purchase a standard training package including a set curriculum, 
and outside instructors. As reported in an early study of consultants and training contractors 
(Moore, Blake, et. al., 1997) many employers are sold on the training package and the ETP 
program by the same consultant. Thus some employers have little exposure to training 
models other than those offered by the training consultant. However, we also found that 
skilled consultants can quickly customize training for individual employers. In several 
examples, consultants or training contractors spent time studying a company’s processes and 
culture, then incorporated that knowledge into the training with examples, tools, or materials 
from the company. 
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A second factor that leads to a lack of customization is outside requirements to use a standard 
curriculum. This requirement may come from a major customer such as Boeing which has a 
standard quality training curriculum it requires (such as ISO 9000), or it may come from a 
corporate parent. Again, if these curricula are not directed at an appropriate level for the 
individuals to be trained or do not provide specific examples from the company’s process, 
they are often ineffective. 

A final factor that has diminished the customization of ETP training is the growth of training 
agency projects, where individuals from many different companies and indeed, several 
different industries are mixed together into a single class. In these situations it is impossible 
even for a skilled instructor to customize the training. For example, an office automation 
class may include technical workers who are there only to learn how to manipulate data in a 
spreadsheet, clerical workers wanting to learn advanced word processing functions, and 
production workers wanting to learn basic database functions so they can maintain inventory. 
Each trainee will be subjected to substantial periods of instruction on topics of little 
immediate value to him or her. We discuss this problem at length in our section of consortia 
and training agencies. 



TAC Aviation: Standard Curriculum Makes for Little Learning 

TAC Aviation is a small, job shop, aerospace company making parts for Boeing 
and other prime contractors. Its ETP training provided VESL, TQM, and SPC 
training to about 50 workers. 

The project operated under an important constraint in that they were mandated by 
their biggest customer, Boeing, to use the Boeing TQM/SPC curriculum. All 
parties agreed the curriculum was too technical and too detailed for the needs of 
these employees. According to the training consultant “the Boeing curriculum is 
a major problem.” For example, managers, supervisors and trainers agreed that 
there were far too many SPC charts covered by the curriculum and that the SPC 
was in general, too technical for the needs of the trainees. There was also a 
perception by all parties that there was too much material to be covered in the 
time allotted. 

Data from our trainees survey showed that while 25% of the trainees reported 
they learned “all of what was taught” in the areas of TQM, and 38% in 
production techniques, only 7% reported that they learned “all of what was 
taught” in the SPC area. 



S Effectiveness of SOST 

We found that the quality of SOST varied widely across the projects we visited. SOST is a 
powerful instructional approach when applied correctly, but all too often we found that SOST 
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activities had limited relevance to training, were poorly supervised, and contributed little to 
the effectiveness of training. 

Good SOST has a number of key characteristics. First, the assignments follow the topics 
covered in a timely manner. Second, the tasks assigned deal with immediate, work-related 
problems. In many cases, we found that instructors assigned generic problems or had 
trainees develop their own problems to work on. For example, in a case where trainees were 
learning problem-solving skills, they were given generic “life problems” such as planning a 
home remodeling project, rather than problems germane to their job. Finally, trainees need 
attention from instructors while they complete SOST assignments. Employees need to be 
able to get help promptly when they “hit a road block” in their assignment, and they need to 
get immediate feedback on the quality of their work so they know that they are using the new 
skills effectively. 

In our fieldwork we found that SOST was often poorly delivered. This observation was 
confirmed by our questionnaire on which trainees across the projects rated the quality of 
SOST as substantially below the overall quality of training (SOST was rated 2.87 versus 3. 1 1 
for training overall). It appears that SOST is added to a number of contracts to increase the 
value of the contract, without increasing the hours employees are off the job and so seems to 
be implemented half-heartedly. Trainees often complained about the seemingly waste of 
time associated with SOST. In interviews about SOST the main topic that came up again and 
again was the difficulty of completing the paper work required to document SOST, and the 
pressure employees felt to complete assignments. Consequently, when we asked them to 
recall their SOST experience, learning was seldom the first thing that came to the trainees’ 
mind. The text box blow illustrates one project in which the SOST component contributed 
very little. 



SOST at Correct Disk: The Experience at an Otherwise Successful Training 
Program OR Another Way to Mark Up the Training Costs 

Correct Disk (CD), a maker of disk drive parts in the Silicon Valley, experienced 
savings of over $5,000,000 from actions and plans started during ETP-funded 
training. CD employed over 1500 workers in California and trained 950 of their 
workers at a cost of $1,700,000 paid by ETP and about $1,200,000 more paid by 
CD. The immediate improvements in performance more than covered the costs 
of training, and the impacts of the training have gone a long way to enable CD to 
survive and prosper in an increasingly competitive and ever more commodity- 
driven market. CD contracted with an industry association for the training, 
which was done on-site by two training consultants. 

In spite of the overall success of the training, the trainees commented that the 
SOST was not effective. The contract called for a total of 68 hours of class, 1 1 
hours of Lab and 107 hours of SOST for which ETP paid $9.69 per hour. From 
our interviews, it seems that the actual time “spent” was 4 hours of SOST and 1 
hour of lab per week for the 1 3 weeks of training. For the SOST component of 
the training the trainees had to devise problems to solve. Trainees reported that 
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the problems became quite contrived and were often not applicable to their jobs. 
It was difficult for the instructors to assign appropriate SOST homework because 
of their lack of intimate familiarity with the company’s processes. SOST took 
time away from more productive work and resulted in trainee dissatisfaction with 
the entire SOST process. Given a 40-hour workweek, the lab and SOST took 
one eighth of the time for one fourth of a year. More than one worker, tom 
between being productive while at the same time trying to fulfill the SOST 
requirements, signed off on the SOST even though the hours were not spent on it, 
so that the contract could be fulfilled. 



Basic Batteries: The Right Way to Do SOST 

In our questionnaires, 56.3% of the trainees at this site rated the effectiveness of 
SOST as “good” and 31.3% thought it was “excellent.” SOST worked well here 
because the company used in-house trainers who knew the production processes; 
the trainer was the union shop steward so the company had the cooperation of the 
union; and, the supervisors were extremely cooperative in helping the trainees as 
they completed their SOST assignments. One trainee we interviewed felt 
encouraged to use the new skills and knowledge due to “the on-site training that 
we received.” 



It is important to note that the projects we studied operated under an SOST policy different 
from the one in operation today. At that time payments were based on trainee hours spent on 
SOST assignments. The policy allowed 10 trainees per instructor for SOST training. Each 
trainee had to document every hour spent on SOST training. The current policy pays only 
for instructor time spent on SOST and only requires that the instructor’s time be documented. 
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As our model suggests, reinforcement of training and generally good management leads to 
significant impact, while no reinforcement or poor management negates the impact of even 
good training. Based on our fieldwork we identified five categories of management 
intervention, which shape the impact of training on the company’s quality and productivity. 
They are: 



S shaping the meaning of training, 

S creating opportunities to use skills, 

S creating rewards and incentives, 

S establishing levels of participation, and 
S institutionalizing training. 

• Shaping the meaning of training before and during training 

Most frontline manufacturing workers have limited formal education, and have often had 
negative experiences in the classroom. Not surprisingly, we found that most ETP trainees 
reported that their first reaction to any announcement of training included anxiety and 
resistance. The uncertainty about what the training will involve, fear of embarrassment in 
front of their co-workers, and worries about changes in their work life as a result of training 
were all reasons given by workers for their initial negative impressions of training. These 
anxieties are not unique to production workers; few middle-aged managers would be eager to 
have their math skills tested either. 

We found that in successful projects, managers anticipated these reactions and worked hard 
to answer the employees’ objections to training. It became apparent as we studied the 
projects that it was important that both top managers and immediate supervisors send a 
positive message about training, along with enough specifics about the company’s plans to 
reassure anxious workers. 

Workers look at training as an event that carries important messages about the future of the 
company and their role in the company. Often these messages can very powerful. One 
worker in a small aerospace company told us, “When I learned about the training program I 
thought: at last... after all the layoffs something positive [is happening], something that is 
moving us forward.” Another trainee said that training told her “...the company cares about 
me and I have a future here.” Our survey data clearly showed that after training most 
employees had a much more positive attitude about the company and their future in it, which 
they attributed to training. 
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Workers can also see a negative meaning in training. Some workers believe that training will 
be used to speed up production at their expense, or that training classes will be used to sort 
some workers into less desirable jobs. Some workers fear that they will be laid off if they 
cannot master the training material. 

Communicating effective, positive messages about the training program often needs to 
involve the use of symbols and ceremonial events in a very direct and public way to show 
employees what training is all about. Companies did this by giving the training programs 
upbeat names, holding dramatic kick-off meetings, having important executives and or union 
leaders address employees about training, or using company newsletters, posters or banners 
to reinforce the key messages about training. These symbolic events emphasized to the 
workers that the company valued the training and their employees. 



Fret Musical Instruments: Shaping the Meaning of Training 

Fret manufactures internationally known musical instruments with a principally 
Spanish-speaking workforce. In 1986 Fret only had about 50 employees at the 
site. After a management buyout, the company grew rapidly and committed to 
producing high-end instruments. 

To achieve the quality level necessary to compete in the musical instrument 
world, Fret knew it would have to upgrade the skills of its workforce 
dramatically. It was faced with training a workforce with little formal education. 
The managers anticipated that employees would be anxious about training and 
resistant to it. Further, the managers recognized immediately that motivating the 
trainees to achieve both goals of completing the training and using their new 
skills on the job would be difficult. Therefore, Management created a program 
called Qual+. The focus of the program is to be a visible symbol of the 
company’s commitment to quality and its employees. Training began with an 
all-company meeting in which the manufacturing vice president introduced both 
the training and the Qual+ program. Interestingly, he did not emphasize the 
value of the program to the company but rather the value to the individual 
worker. “I told them this was their chance to improve. They could learn 
valuable skills that would help them here, in their personal lives, or in another 
job.” 

“I measure the success of the program by the smiling faces when employees 
become members of the Qual+ team.” “We use the training not just to build 
skills but to build a sense of membership and family at Fret,” says the Vice 
President for Production. Training is tied up with the Qual+ program in that 
when employees complete training there is a formal ceremony where graduates 
get a pin and diploma, which makes them a member of the Qual+ team. The 
only meeting room in the guitar factory is called the Qual+ room. In our 
fieldwork we found that Fret’s employees valued their membership in Qual+, and 
bought into the idea that the problem solving and decision-making skills learned 
in training benefited them and not just the company. 
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• Creating opportunities to use skills 

Newly learned skills have a very short shelf life. If employees do not put the skills learned to 
work soon after training, most of what was learned will be forgotten and trainees will grow 
cynical about the value of training. Effective programs make sure trainees have the 
opportunity to put skills to work during or immediately after training. For example, one 
small manufacturer of printed circuit boards had a policy that as trainees moved through the 
course in printed circuit board design they were systematically assigned more difficult tasks. 
As they neared the end of the program they were allowed, for the first time, to talk directly to 
designers and engineers at client companies to resolve design problems. Each step allowed 
trainees to use newly acquired skills and gain prestige in the eyes of their co-workers and 
customers, cementing the learning that had taken place in the classroom. 

We saw another example of how management can reinforce the use of skills immediately 
after training in a small aerospace supplier. As soon as the training was over, management 
formed quality teams, and created a process where teams could request time to meet to work 
on problems. When the teams generated plans for quality improvements they were allowed 
to make formal presentations to a management committee. If the committee accepted the 
plans, resources were allocated for the process changes and the team was given public 
recognition. During training, employees were told about the system and how it would work. 
Management’s quick action on the first proposals from quality teams sent the message that 
the training and the quality program were for real. 

We also saw some examples of what goes wrong in this process. We found two major 
barriers to the immediate reinforcement of skills after training. The first is that necessary 
technology or programs are not in place when training is completed. 

The second major barrier is the lack of buy-in from line managers and immediate 
supervisors. While top management often works hard to “sell” training to frontline workers 
they often overlook first line management and supervisors, whom they assume will see the 
value of training. We found several cases where supervisors, who had to struggle to maintain 
production while workers were away at training, became very negative about the training and 
did little to reinforce it once workers returned. Part of the resistance to reinforcing training 
was the simple fact that implementing new techniques or programs initially slows 
production, which was already behind schedule. In essence, these supervisors were put in a 
double bind of having to maintain production while taking resources away from production 
for training and implementing production innovations. 
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• Creating rewards and incentives 



Companies we visited held widely varying views on how to provide incentives to employees 
to get them to complete the training and take it seriously. We found we could split the 
companies’ strategies into three categories. 

^ Explicit incentives : systems where employees receive immediate rewards for 
completing training. These rewards come in the form of guaranteed promotions 
or pay raises, or at least the eligibility for promotion. This approach is illustrated 
in the text box on Basic Batteries. 



Quid pro Quo at Basic Batteries: Explicit Reinforcements to Training 

Basic Batteries (BB), located in Southern California, is a major producer of lead- 
acid batteries primarily for automobile and marine use. They have three facilities 
in the LA area. The ETP project studied was in the main manufacturing facility. 
The batteries produced are sold under a number of private labels. About half the 
workers are Hispanic and half are Vietnamese. BB has long had an interest in 
employee training and development; for example they began providing ESL 
training in 1991-92. BB workers can make an above average living for the semi- 
skilled work they perform and as a result turnover is very low. BB is certified for 
QS 9000, which is stricter than ISO 9000 in order to satisfy Original Equipment 
customers. 

In the ETP contract all training was done in-house with in-house trainers. A 
consultant trained the trainers. Their goal was to reduce costs, especially through 
lower scrap rates. They were successful: they reduced scrap rates by 50%: from 
over 4% to 2% of cost of goods sold. 

To reinforce the importance of training, halfway through the training, BB 
instituted performance-based pay for key activities valued by management. The 
idea was that the workers would value the important ideas of training more if 
they were paid for achieving them. Average wages were $13 per hour, plus 
benefits. This was above average pay in this area considering that workers only 
need to be semi-skilled and semi-literate. Nevertheless, management determined 
that if the important lessons of training were to be acted upon, they still needed to 
emphasize their importance by offering monthly bonuses. The monthly bonuses 
could add up to about $250, in the form of “Sam's Dollars” redeemable at Sam’s 
Club or Wal-Mart, for meeting a variety of goals. The first was relatively 
simple: $50 per month for attending weekly health and safety meetings. They 
could receive another $50 a month for productivity and quality above overall 
monthly plant productivity. They could receive $5 per day as a daily individual 
productivity bonus and an extra daily bonus of $10 if they exceed 5650 batteries 
per shift. 
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^ Implicit incentives-, systems where trainees expect they will eventually be 
rewarded for completing training but the timing and nature of the reward is not 
explicit, as illustrated in the cases of Sports Brace and BT industries. 



Sports Brace and BT Industries .* Implicit Incentives for Training 

At this company managers do not have an explicit policy of giving employees a 
raise or a promotion when they complete training, but we found unstated 
“implicit” expectation of both mangers and employees that there would be 
rewards from training. Both companies were served by training agency consortia 
which uses ETP funds to train machinists in CNC technology. 

Sports Brace is small dynamic company that makes knee braces and a few other 
healthcare related products. They appear to encourage employee initiative. If 
employees are able to do more, or innovate they are encouraged and rewarded. 
We interviewed two trainees; both seemed to work with very little supervision 
and both talked with pride about being recognized for being able to do new and 
more complex tasks after completing training. 

One machinist reported he was consulted on the purchase of a new $500,000 
CNC machine. The company bought the same brand and type he had been 
trained on at NTMA, in part so he could be more productive right away. The 
second employee reported that he was given “more interesting work to do,” and 
received a pay raise. Both employees insisted that they had not been promised 
any explicit rewards when they entered training but because of the culture of the 
organization they were sure it would pay off in the long run. 



S No individual incentives: some employers are opposed to the idea of individual 
rewards. Often they see training as a company- wide effort to improve rather than 
the development of an individual. Others simply have pay systems that tie pay to 
job categories and seniority and don’t want to violate the system with special 
incentives for training. The Fret instruments example below reflects this view. 
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Fret Musical Instruments: No individual incentives 

Interestingly Fret, which put a great deal of effort into defining the meaning of 
training, did not believe in explicit individual rewards. The company, while 
nonunion, ran on a very traditional compensation system. Individuals were paid 
based on their job classification and seniority only. Management was committed 
to a clear, orderly, compensation system and believed that special raises or 
incentives would disrupt it. 

The production manager believed strongly that being allowed to work on 
interesting problems, spend time with managers and engineers, and receive 
recognition created enough intrinsic motivation to sustain the quality program. 



^ Disincentives: We also found that managers sometimes inadvertently create a system that 
punishes workers for participating in training and thus provides negative reinforcement. 
For example, we found instances where trainees returned from training to find angry and 
hostile supervisors who had scrambled to cover for them while they were absent for 
training. 

• Establishing levels of participation 

One question this study tried to answer was: Is there an optimum level of participation in 
training programs? We explored this issue and found that the needs of companies varied so 
widely that no general conclusions could be drawn about participation levels. In some 
companies, training a handful of specialists was viewed as very valuable by employers. 

Other employers deemed it essential that all employees be trained. We did encounter one 
important finding about participation in quality programs that bears some exploration. 

The conventional wisdom in most TQM/SPC-type quality programs is that all employees 
should be involved. This was reflected in many of the TQM/SPC projects we studied in that 
the projects trained all employees in these techniques. Yet, in interviewing managers we 
found a strong belief among many of them that they could achieve significant improvements 
in quality with only a minority of employees participating. One manager at Fret Musical 
Instruments described the dynamic this way: 

“Thirty percent think the training’s crap; 30% just don’t care; and 30% really get it 
and will lead us to substantial improvement.” 

In general, this manager does not buy the TQM model where all workers are on teams and 
involved in quality improvement. Rather he thinks about 25% of the workers has the skills 
and motivation after training to identify and solve problems, and he wants to work with them. 
He says: “I give motivated people something meaningful to do. I give them praise and time 
to work on their problem with management.” So membership in quality teams is voluntary 
and no cash bonuses are paid to groups or individuals for solving quality problems. This 
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company reported dramatic increases in quality improvements yielding substantial cost 
savings, and the company is currently making a major expansion, lending considerable 
weight to these views. 

We found several other companies where, after training everyone in TQM techniques, the 
management made membership in teams voluntary and again reported significant quality and 
productivity improvements. In general managers seemed to believe that innovations would 
only come from employees who voluntarily took on quality problems. Other employees who 
wished to remain in a traditional production workers role should be allowed to do so. This 
contradicts earlier research particularly from the auto industry (Wilms, 1996), where 
universal participation in quality teams was deemed essential to a program’s success. The 
implication for ETP seems to be that different production systems likely require different 
levels of participation. Employers are likely to be the best judges of how much participation, 
and thus how much training, is enough. 

• Institutionalizing training 

A goal of ETP is to serve as a catalyst for training. The idea is that if companies have a 
successful experience with ETP training those companies will be more willing to invest their 
own money in additional training. We found in our fieldwork that successful training did, in 
fact, lead to additional investment in training. We have already reported some examples of 
this phenomenon; see for example the box on Flow Pumps. 

One observation we did make was that the more involved in-house staff were in the training, 
the more likely it was that the training would be institutionalized. The logic of this is simple: 
if the company owns the curriculum and has trained instructors easily available, it is likely to 
continue training new employees or upgrading existing ones. Continual training is less likely 
to happen if they must hire outside consultants. 

In the case of training agency projects, which run classes away from the work site with 
trainees from various companies, we found a different type of institutionalization. In these 
cases we found that if companies sent an employee to the training and had a positive 
outcome, they would send additional trainees. Eventually participating in the training agency 
training becomes built into the human resource practices of the company. Again, the logic is 
apparent since this particular ETP training is off-the-clock, does not disrupt production, and 
is free; there is virtually no cost in institutionalizing the training. 



Techno Tubs: Integrating Training Agency Training into Organizational 
Development 

Techno Tubs is a small manufacturer of lightweight hot tubs made out of foam 
rubber. When a new CEO arrived five years ago just one employee was 
computer literate. To catch up to the competition, the new CEO launched a drive 
to computerize the company’s paper-based marketing, finance, customer service, 
and personnel systems, and to install CAD/CAM in its production areas. The 



O 

ERIC 



ETP At Work 



43 



34 



company began sending from one to three employees a session to the ETP 
Training Agency office automation training program and continued at that rate 
until a sufficient number of employees were trained. Both the company and the 
employees were pleased with the results of the training. Techno Tubs 
subsequently built an office-automation, computer-literacy requirement into its 
promotion process so that people in many of the company’s departments do not 
progress beyond a minimal level without these skills. Workers now attend office 
automation training as needed in response to evolving industry practices, normal 
worker turnover, and company expansion. 



Flow Pumps: Successful ETP Project Leads to Permanent Training System 

Flow Pumps is the pump manufacturer described earlier. Prior to the ETP 
training the company had no formal training. Their ETP training project led to 
substantially increased quality and a positive change in the organization’s 
culture. Because of these positive outcomes the company established a 
systematic training program for both existing employees and new hires. The 
company now has two permanent classrooms. They consider some of the 
courses developed under ETP “core to the company’s culture” and all new hires 
take them. They include basic math, SPC, problem solving, open book 
management (how to read company financials), and interpersonal/team skills. To 
reinforce the importance of training, the company created a system where 
employees must complete the training and pass tests tied to the training in order 
to get promoted. 

One important reason why training became institutionalized was that early on 
managers decided to use in-house trainers for most of the training. Their trainers 
were managers and quality engineers who received some “train-the-trainer” 
training. Early on there were a couple instructors supplied by Fresno Community 
College, but they played a minor role, and taught the first classes of basic math 
and safety. Another factor supporting the institutionalization of training was that 
many of the in-house instructors enjoyed the teaching experience. In fact, Ralph, 
the company president and an engineer by training, taught basic math. This was 
a powerful experience for him. Reflecting on the experience he noted, “I’ll never 
make fun of teachers again; it’s a tough job.” He was delighted with the 
outcome, however. Now he knows a number of production workers well, and 
they know him. He continues to teach sections of the basic math course. 
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V Impact of ETP-Funded Training on Company 
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Introduction 

ETP-funded training has a positive impact on companies in almost all cases. In this section, 
we first discuss the importance of ETP-funded training. Second, we discuss the issues 
regarding measuring the impact of training on performance with respect to the availability of 
information and the case study method of assessing the performance impact of training. 
Finally, we present findings from our fieldwork about the impact of ETP training on the 
companies we studied. 

• Why ETP-Funded Training Matters 

Companies that qualify for ETP-funded training face out-of-state competition, are creating a 
high performance workplace, or are diversifying product lines, among other challenges. 
Companies and workers that face out-of-state competition are in a position to benefit from 
increases in their human capital brought about by ETP-funded training. This should lead to 
more creative and higher-performing workplaces that remain competitive, through enhanced 
worker productivity. Given the increasingly competitive markets, those producers who can 
provide products at competitive prices and survive, do so because they are able to produce 
those products most efficiently. Productive efficiency is increasingly dependent upon human 
capital workers’ skills, knowledge, and attitudes. The investment in human capital explains 
how the U.S. can compete in global markets where the cost of labor in some countries may 
be only a tenth of labor costs here. It also underscores the increasing importance of creating 
a skilled and productive workforce. Higher wages and living standards for California 
workers who compete in a global market will only come about because of a greater 
efficiency and quality of the workforce. California workers must do it better and more 
efficiently if they are to enjoy improving standards of living. 

From a policy point of view, ETP-funded training is like an investment tax credit. An 
investment tax credit encourages businesses to adopt the latest production technology by 
subsidizing the purchase of new equipment through a tax credit. This subsidy recognizes the 
benefit to the workers and to the economy from having more modem and efficient production 
facilities. Likewise, state-subsidized training also acts as a catalyst to enhance the capital and 
technology employed in production. Without skilled and quality-minded workers, even 
abundant capital and cutting-edge technology cannot be efficiently used, which eventually 
leads to reduced returns, squandered potential growth, and lowered standards of living. 
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ETP -funded training also acts as a catalyst in the sense that companies use it to jump-start 
their training efforts and effect internal culture changes. We have observed that the TQM 
training has been effective in changing work place cultures by empowering the workers and 
making quality and problem-solving part of the culture, part of everyone’s responsibility. 
Without the ETP-funded training, many of the companies we interviewed would not have 
undertaken the ambitious programs on the scale they were implemented. We believe that the 
profound workplace changes achieved were enabled by the scope and scale of the training, 
something which would have been out of reach without ETP funding. Thus, in many cases 
ETP funding was instrumental in creating training programs on the scale necessary to effect 
real cultural changes in the workplace. 

ETP-funded training acts in another way to foster change. ETP contracts bond management 
to change because of the third-party accountability. Writing up the applications and contracts 
usually causes management to think through the training process in advance, giving it a 
higher probability of success. The reporting and performance monitoring by ETP also 
increases managerial involvement in the process. This third-party accountability strengthens 
management follow-through, making the probability of long-term success higher. 



Measuring the Impacts of ETP-Funded Training 

We set out to gather detailed financial and operational data for each company and for the 
industry in which the company operates. With this, we expected to compare the performance 
of the company before the “treatment” of training with its performance after the “treatment.” 
We could also compare the company’s performance with its industry peers both before and 
after training. This kind of analysis would have allowed us to assess the impact of training 
on the company and to evaluate the value of the benefits of training. Given the detailed 
industry data we had available, a cross-sectional analysis seemed to be a reasonable approach 
to assessing the impact of training. 

Unfortunately, once the fieldwork began, we found that we could not get the detailed 
information we desired, at the company level. In some cases companies were unwilling to 
provide detailed financial data. In other cases, the unit trained was a small part of a much 
larger organization so that training could not reasonably be expected to affect company-level 
financial data. However, we also found that there were non-monetary benefits that we could 
ascertain in the site visits. Thus, we decided to use the case study method of analysis 
because it would allow us to capture the multi-dimensional impacts of ETP-funded training 
on the company. Case studies also provide the flexibility to come up with performance 
measures on an ad hoc basis, using whatever information was available. Our findings from 
this fieldwork will be tested later in the study when we examine financial records on all 
companies served by ETP during 1994-95 and 1995-96 in a large-scale statistical analysis of 
ETP projects. 
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• Barriers to Measuring the Performance Impacts 

There were many challenges to measuring the performance impact of training. First of all, 
most companies did not set measurable goals for training, and therefore were unprepared to 
share with us how the training they received impacted their companies. They simply did not 
think of training in a way that related to measured financial performance. The managers just 
assumed that training should make a difference and therefore should be implemented. 

Second, many companies surveyed are private and simply did not share financial or operating 
performance information. Also, since managers view performance in many ways, there was 
no single performance measure that could be applied to all companies. The public 
companies we surveyed were so large that the detailed financial information they did provide 
did not show performance impacts from training because the number of individuals trained 
was so small relative to the whole. Because of the lack of detailed and uniform financial 
information, we could not assess the impact of training by comparing industry performance 
benchmarks to our companies. 

• Rationale for Using the Case Study Method 

Given the lack of consistent performance data, we had to customize our analyses to the 
unique types of information provided by each company. Thus, the analysis of the impact of 
training on companies had to be done on an ad hoc basis. As a result, case studies have 
become the best way to convey our findings. The lack of detailed, standardized data forced 
us to become creative and flexible as we proceeded with the detective work of assessing the 
impact of training on companies’ performance. We were able to gather the information 
needed to construct the case studies by making site visits. Managers told us how they viewed 
performance, and we tailored our questions to at least partially, determine training impacts on 
the company. The advantage of the case studies over statistical cross-sectional analyses 
allows us to see the different ways ETP-funded training impacts company performance. 

Our efforts paid off. We were able to find clear evidence that training had a significant 
impact on some companies, and we also found evidence that training had little impact on 
others. Our ability to ferret out information during the site visits gave us confidence in the 
validity of this type of analysis, and in our results. If there were benefits to training, we were 
able to measure them in some way. When we could not find evidence of an impact on the 
company, it was because most likely, there was little to none. 

• Overall Impact of Training on Trainees 

Before we look at the impact of training on individual companies, we want to explore how 
trainees, as a group, perceive the impact of training. In the methods section, we described the 
evaluation questionnaire that we administered to as many trainees as possible at each site 
visited. On the evaluation questionnaire, we asked each trainee three questions to assess the 
impact training had on them: 
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1 . How often did they use the skills they learned in training? 

2. What impact did training have on their individual productivity? 

3. How had their work environment changed since training? 

• Skill Use 

For training to have an impact on productivity, trainees must use the skills they learn in 
training. Ninety-five percent of trainees report they use the skills they learned in training at 
least occasionally. As Figure 3 shows, slightly over half the trainees reported that they used 
at least some of the skills they learned in training everyday. For example, office workers 
trained in office automation are likely to report that they use the software applications they 
learned everyday. About a quarter of the trainees report they use the skills they learned once 
a week or more. One example of this response came from a production worker who had 
TQM training and is now a member of a quality team that meets weekly to work on 
problems. Seventeen percent said they use the skills less than once a week. An example of 
this type of skill use is the production worker who learned SPC techniques, but who only 
uses it occasionally to deal with specific quality problems as they occur. Only a very few 
trainees, 5%, report that they never use what they learned in training. As we noted in the 
section on management reinforcement, one reason why skills are not used after training is the 
lack of management action to ensure that reinforcing processes are in place when trainees 
return to the job. For example, a trainee who receives SPC training will not use the skills if 
an SPC system does not exist on his or her production line. 



Figure 3 

How Often Trainees Use Skills Learned 

Everyday 
Once a Week or More 
Less than Once a Week 
Never 
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We also examined which types of training trainees reported using most often. This analysis 
is restricted to skill areas where we had at least 60 respondents. As Figure 5 shows, skills 
used more frequently than average included skills that related directly to improving 
production, TQM and Production Techniques, and basic computer skills including MS Office 
and General Windows. Skill areas used least frequently tended to be more technical such as 
SPC. 



• Impact of Training on Productivity 

Overall ETP trainees believe that their training has led to higher productivity. As Figure 4 
indicates, two-thirds of trainees reported that training led to either a “substantial increase” or 
“major increase” in their productivity. Slightly less than a quarter reported a “small 
increase” and only 9% said training led to “no increase” in productivity. 



Figure 4 

lm pact of Training on Productivity 

(Reported by Trainees) 
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• Differences by Type of Training 

Using a regression analysis we identified the factors that were associated with trainees 
reported increased productivity. Clearly, the more of the material trainees mastered and the 
more they used what they learned, the greater the impact it had on their productivity. 
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Figure 5 

Relationship of Amount Learned, Skill Use and Impact on 

Productivity 




B Amount Learned ( 4 Point Scale) ■ How Often Used (4 Point Scale) □ Impact on Productivity (4 Point Scale) 



A similar pattern occurs when we look at the amount learned, skill use, and impact of 
different types of training on productivity. Figure 5 shows the types of training most 
commonly found in the sites we visited. (We included all categories of training with more 
than 60 respondents). We then ranked the types of training based on the impact of 
productivity reported by trainees. The graphs reflect the larger pattern that as the amount 
learned and skill use increases, so does the impact of training. 

In addition, we can see that three types of training had a greater impact than training overall. 
The results reveal that the types of training rated as having the highest impact were 
“Production Techniques” and “TQM,” both of which deal directly with improving quality 
and productivity. The only other type of training that came in above the overall figure is 
training in MS Word, a very commonly used skill. These are also areas where trainees rated 
the amount learned and skill use above the average rating. 
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Figure 6 

Trainee’s Rating of Changes Since Training 



Feel Mare Mativated and Involwsd 
Mare Positi\e Attitude About the Co. 
Better Communication Between Mgmt and Workers 
Feel More Optimistic About Future In Co. 

Less Stress 
Better Relations Between Workers 
Workers Treated Better 




Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, 4 = Strongly Agree 



Impact of Training on Work Environment 

As we began our fieldwork we were struck by the fact that when we asked trainees and 
managers about the impact of training they talked first about the qualitative impact of 
training. Rather than talking about scrap rates, or labor productivity, they talked about 
improvements in communication, motivation, or stress reduction. As Figure 6 indicates, 
trainees report dramatic improvements in many aspects of their work life. More than half the 
trainees surveyed “strongly agreed” that they “felt more motivated and involved at work,” 
“had a more positive attitude about the company,” and “there is better communication 
between management and workers” since the training. These results confirm what we saw in 
the field. The decision to provide training is a powerful symbol which reveals that it is 
important for management to shape the meaning of training, so that trainees see it in a 
positive light. 

We would expect these types of positive qualitative outcomes from TQM and related 
training, which focus on improving communication, reducing hierarchical barriers, and 
involving frontline workers in decision making. So we were surprised when we found 
similar positive results for technical training as well. It appears that simply the act of 
investing in training workers makes them see the work environment in a positive light. 
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We also found a relationship between these qualitative outcomes and reported increased 
productivity in our survey data. A regression analysis revealed that the more strongly 
workers agreed stress had been reduced, or that they felt more motivated, or that they had a 
more positive attitude towards the company since training, the greater increase in 
productivity they reported. 

Overall our regression showed that the changes in workplace environment caused by training 
questions were significant predictors of worker reported changes in productivity. The results 
indicate that the positive changes in work environment variables accounted for almost 40 % 
of the variance in the change in productivity variable. The table below shows that 
“experiencing less stress on the job after training,” “feeling more motivated and involved,” 
and having a more positive attitude about the company” are variables that were significantly 
associated with increases in productivity. The data does not allow us to assume that changes 
in the work environment variables caused increased productivity, but they do indicate that 
when workers report an improved working environment, they also report improved 
productivity. 



I 



workers since training 



Less stress since training 


.228 


.002 


Workers treated better since training 


-.032 


.663 


Better relationships at work since training 


.132 


.064 


Feel more motivated and involved since 
training 


.242 


.003 


Feel more optimistic about my future in the 
company since training 


.067 


.403 


More positive attitude about the company 
since training 


.155 


.037 


Adjusted R square 


.397 
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Better communication between supervisors and 



.014 
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Impact on Company Performance 

• Cost-Benefit Approach to Evaluating the Impact of ETP-Funded Training 

An evaluation of the impact of ETP-funded training should attempt to identify and measure 
the benefits of the training. The benefits of training cannot be judged without a context; so 
we decided to use the cost of the training as a benchmark to judge the effectiveness of the 
training. As stewards of the taxes paid by the citizens of California, ETP needs to determine 
whether the benefits justify the costs of using taxpayers’ money. The cost-benefit approach 
is the basis of all business investment evaluations. 
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As we proceeded with the fieldwork, it became clear that analyses of the benefits of training 
would have to be tailored to the company studied. As we noted earlier, most companies did 
not set concrete goals for training, and as a result, did not systematically track the impact of 
training on the company. This lack of tracking the performance impact reflects not only the 
difficulty of measuring performance impacts on the company, but also management’s belief 
that training is beneficial. 

Because companies either did not keep specific records documenting in quantifiable ways the 
effects of training, or else when they did, they were idiosyncratic to the company, we found 
that the case study method would be the best way to observe the impact of ETP-funded 
training on company performance. We also realized that assessing the financial impact of 
training would entail persistent detective work, questioning managers and workers in creative 
ways, because there was no universal performance measure or impact factor that could be 
used for all companies. It also became clear to us that the financial impact of training may 
not always show up on the corporate “bottom line” because the cost-savings from training 
could underwrite higher pay, better employee benefits, better quality products, better work 
environment, job creation or retention, or simply ensure company survival. Thus, it would be 
simplistic and inaccurate to think of the financial cost savings as “corporate welfare” that 
would be reflected in profit margins. 

W e also found that most of the private companies were reluctant to share financial 
information, so we had to work with piecemeal and incomplete financial information in the 
form in which they were willing to share. Sometimes this was in the form of reduced scrap 
rates, cost savings on a particular line, production increases for a product, or lower levels of 
rework. Even for publicly owned companies for which accurate financial information is 
readily available, we were not able to find a measurable impact by examining consolidated 
financials. This is because they are so large that the training would have no observable effect 
on the consolidated financial statements. So even with these companies, we had to work 
with anecdotal production and financial data. 



Stand-Alone Training 

The training in stand-alone contracts was more likely to be customized to the needs of 
the companies. As a result, the impacts were greater and longer lasting than those associated 
with consortia or training agency training. We also observed that when the trainers were “in- 
house, the impact tended to be greater, and it was more likely to lead to further follow-up 
training. 
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• TQM Training improves quality and reduces per-unit costs 



Basic Batteries 
619 trained 
ETP Costs: $500,876 
Estimated Benefits: 

■ Reduced Scrap: $1,230,000 

■ Additional Cost Reductions : $ 1 ,040,000 

■ Cost-Benefit ratio: over 450% for the first year 



BB is a major producer of lead/ acid batteries primarily for automobile and marine use. They 
have three facilities in the Los Angeles area. The one we visited is the main manufacturing 
facility. Its major competitors are large multinational companies. They produce under a 
number of private labels. Average wages were $13 per hour plus benefits. This was above 
average pay in this area considering that workers only need to be semi-skilled and semi- 
literate, and as a result, there is low turnover. BB kept detailed records of quality and 
productivity. By looking at the performance before and after training, we can infer the 
impact of training on the company in dollar terms. 

BB had on-site training using in-house trainers dealing with SPC and TQM. Six hundred- 
nineteen workers were trained. This company achieved both scrap rate reductions and unit 
volume production increases resulting in a benefit to the company of about $2,270,000 per 
year. Before training, daily production was 8500 units per day and after training, production 
rose to 10,300 units per day. This 21% increase was not due to greater investments in 
machinery or labor, but it reflects an increase in operational productivity since training. They 
are saving about $250,000 in “assembly junk:” QC tests destroy only 2% of the units tested 
versus 38% before training, due to the TQM emphasis. Scrap costs have declined from $0.61 
to $0.27 per unit. These savings results in daily savings of $3,502 and $1.23 million per year 
based on a 350-day work year. This is about half of the total cost savings achieved overall. 
Overall manufacturing costs per unit are now $6.87 versus $7.50 in 1994, and this is 
expected to drop further, to $6.34 in 1999. The average current production currently runs 
10,300 units per day. This translates into savings of $6,489 per day and $2.27 million per 
year based on a 350-day year. 

Compared to ETP’s investment in training of $500,876, the cost-saving improvements of 
$2,270,000 have yielded a 453% cost-benefit ratio in the first year after training. Even if 
three-fourths of the benefits are attributable to other factors, the return on investment is over 
100%. Moreover, unit production rose 21%. 
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Fret Instruments 

350 of 350 workers trained 
ETP Costs: $677,968 
Estimates of Benefits: 

■ Scrap Rates declined 95%, 

■ Rework savings of over $400,000 per year, 

■ Subassembly efficiency increased from 63% to close to 100%, 

■ 20% planned overproduction for scrap eliminated, 

■ Over 300 jobs created, 

■ Hundreds of jobs retained in Southern California, 

■ Cost-benefit ratio: At least 1 00% per year 



Fret is a premier manufacturer of musical instruments and accessories. In the 1980s the 
company underwent a management buyout, and since then growth has been rapid. From 
having only about 50 employees at the Southern California site in 1986, Fret has grown to 
over 400 employees today. In addition to making musical instruments, the company has 
added an electronics factory to make amplifiers. Fret considered moving its operations to 
Texas or Tennessee in the late 1980s, but a “Red Team” from the state got involved with the 
company and they decided to stay. It was during this involvement with the “Red Team” that 
Fret learned of ETP and decided to do ETP training. Fret used their safety training 
consultant from Systems Management to design the program and to go through the ETP 
application process. The training consisted of TQM and SPC modules. This first ETP 
training was a success, and Fret has another ETP-fiinded program underway at this time. 
Trainees were to come from managerial, supervisory, and hourly workers. 

The training obviously had a significant impact on the company. The production manager 
commented that the training improved people’s ability to solve problems: “Now they have 
tools to bring closure to problems.” He also noted that there is improved communication 
between departments. The trainees had many positive comments. An instrument tuner 
claims that scrap rates declined from 12 racks of instruments every 2-3 days to one rack a 
week, approximately a 95% reduction. A leadperson commented that they went from “63% 
efficiency in sub-assembly up to 100% in the past 8 months.” Interestingly, trainees 
commented that they miss the overtime they got for reworking bad product. Another 
commented that he had “developed a system to avoid rework.” The head of instrument 
production had the most revealing information regarding the benefits of training. He said 
that rework was 25-30% before training and now it’s declined to 5%. They have only one 
person dedicated to rework instead of four. On average it takes 40 minutes to do rework at 
$40 per hour. Assuming the more conservative improvement of 25% to 5% and the current 
production of 325 instruments per day (actually lower than in the recent past) this works out 
to a daily savings of $1,734, a weekly savings of $8,671 and an annual (50 weeks) savings of 
$433,550, just on rework. He also commented that they no longer schedule a 20% 
overproduction in order to have enough good instruments. 
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To summarize, there was at least a $433,500 per year savings just from reduced rework. This 
does not include the obvious increases in other areas of production efficiency, such as 
improved scheduling and higher output. It would not be unreasonable to assume that other 
savings were at least as great as those from rework (cf. Basic Batteries); so it can be 
reasonably estimated that savings were on the order of $800,000 per year. These benefits 
will continue well beyond the first year after training, increasing the impact of ETP training. 
In addition to this benefit, Fret stayed in California, providing several hundred jobs that 
would otherwise have been lost at a great expense to the local economy. 

Compared to the investment of $667,968 by ETP, the benefits have been enormous. By our 
estimates, the annual benefits probably exceeded the one-time cost of training. Finally, Fret 
not only retained jobs in California, but also created new ones when they moved their Oregon 
production here to capitalize on the trained workers. 



• Training Triggers TQM Innovations Throughout the Plant: Works in both “Low 
Tech” and “High Tech” Companies 

We were most impressed with the universal applicability of TQM training funded by ETP, 
and the power it has to change the culture in companies. The training seems to catalyze the 
essential prerequisites for producing high quality products. It fosters an interest in quality 
and instills the attitude of problem solving, independent thinking, and collaboration to do 
things better. The impact is significant whether the company is high tech or low tech, large 
or small. 



T-Bar 

Trained 55 of 55 workers 
ETP Costs: $118,276 
Estimate of Benefits: 

■ Sales, employment and quality increased dramatically 

■ Sales increased from $6 million to over $20 million 

■ Achieved widely-recognized quality certification from Case, the global heavy 
equipment manufacturer. 



T-Bar, located in Fresno, is a small manufacturer of roll bars, tractor cabs, and bulldozer-type 
blade extensions. The products are decidedly low-tech and unglamorous, but now show up 
all over the U.S. and the world. The company did not measure the impact on training in any 
way. However, upon discussing the matter with them, there was clear evidence that the 
TQM training they received had an important impact. The workers meet monthly in quality 
and problem solving groups, and they have representation at top management meetings 
where they bring their suggestions and requests which get moral and material support. One 
worker showed us years’ worth of TQM meeting notes and commented that they were able to 
organize and get storage units for inventory that formerly laid around in the dirt. Workers 
are able to get the equipment they need to do their jobs better. They train their new workers 
in a team-like manner. They report that the TQM and SPC training helped them qualify for a 
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quality certification from Case. This well-known certification qualified them to do work for 
other customers. They wrote a new ISO 9000-based quality manual and now have an in- 
process inspection. They passed Case’s audit and the training helped them pull this off. The 
training set up momentum and a structure that has gotten them lots of work. Being certified 
by Case is a valuable credential. They are also SQA approved. Workers meet twice monthly 
in TQM teams and communicate much better with managers, especially since each team has 
a representative on the overall company team which includes the top managers. They are 
able to pass on ideas and get new tools and processes adopted quickly. 

Sales have more than tripled since training: $6 million to $20 million with a concomitant 
increase in well-paid workers. 



Correct Disk 

Trained 463 of 1,600 workers 

ETP Costs: $834,051 

Estimate of Benefits during Training: 

■ Over $5 million saved 

■ Reduced down time in tooling changes: $500,000 

■ Reduced machine costs of over $4 million per year 

■ Reduced time for one process results in over $650,000 additional 

revenue per year 

■ Immediate cost -benefit ratio: over 400% (not including long-term 

repeated gains) 



Correct Disk is a large, publicly traded Silicon Valley company making state-of-the-art 
components for computer disk drives. It truly competes head-to-head in the global 
marketplace. ETP-funded training had an impact both among production workers and 
professional engineers. 

Correct Disk used ETP-funded training to train 463 workers on-site in SPC and TQM. 
Immediate savings due to training were in excess of $5 million dollars. The full-time TQM 
coordinator supplied us with the following direct quotes from TQM teams regarding what the 
they achieved during training: 

• Reduced HGA inventory discrepancies from 16% to 9% by Oct. 31, 1995; 

• Reduced the average monthly Comptech tooling shortages which 
resulted in greater than $500,000 annual savings in reduced downtime 
by Dec. 6, 1995; 

• Reduced R2A-D defects from 5.4% to 2.7% by June 2, 1995; 

• Reduced Phase 1 MR Slider Fab from 2% to 0.5% by fiscal quarter 3, 

1995; 

• Reduced scrap due to plating parameter input errors by May 1, 1995 - 
went 1 1 weeks without an error and have maintained a lower error 
rate; 
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• Reduced cost of HGA by 1 4% by end of Fiscal Year 1 995 ; 

• Reduced cycle time for pallet treatment by 44% in RRC: from 136 
minutes to 76 minutes. This would result in $18,000 savings in setup 
of RRT and even more by reducing treatment time; 

• Reduced “material starts” from 62% to 40% by Aug. 24, 1 995; 

• Reduced down time on the Mini JIT UV cure systems from 0.4% to 
0.0%. Assuming the downtime created a bottleneck in output, this will 
result in additional revenue of $667,000 per year; 

• UPD went from 97.4 to 1 1 6.7. “This equates to savings of $864,000 
per quarter and $4 million in machine costs.” 

This company had made TQM a part of its culture. This is reinforced by the fact that even 
years after the training there continues to be a TQM coordinator who tracks the impact of 
TQM practices and follows up with training and encourages TQM teams to meet to solve 
problems. 

The immediate results of this training amount to over $5,000,000. This compares favorably 
to the $834,05 1 cost of the training. The long-term effects of training would amplify the 
benefits even more. 



• Training can fail even in a high tech, global leader located in Silicon Valley when 
Management does not support the training 

We observed that managerial involvement and support are absolutely vital for a successful 
training experience. Without top management’s involvement, the workers are unable to 
apply the training and the time and money spent is mostly lost. Managers need to carefully 
assess whether they want to invest in training and then they have to plan for it. The 
following example shows what happens when management pulls support for the training. 



Wavelength 

Trained 212 of approximately 365 employees 
ETP Costs: $115,029 
Result: “Catastrophe” 



Wavelength stands in stark contrast to Correct Disk. This company makes a wide variety of 
high tech electrical devices for commercial and defense purposes. Formerly, Wavelength 
was primarily a defense contractor, but the reduction in military spending in the early 1990’s 
forced the company to change. 

The training began as a consortia type and then went to a stand-alone contract. The training 
was, in the words of the vice president of HR and Administration, “a catastrophe.” This on- 
site training served about 212 of its approximately 365 full time employees. After about a 
week, the managers stopped attending the training and never supported it. In fact, there were 
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reports that managers opposed the TQM approach. “Follow-up” came in the form of 
managers cursing and threatening the workers if they didn’t catch up on work missed due to 
training. Nothing was implemented as a result. In fact, the training had to be stopped for a 
while because of the disruptions in production, some of which was attributed to excess 
paperwork, SOST, and “homework” associated with the training. Overall, the company was 
worse off because of the training. The workers were discouraged because they had learned 
valuable skills only to be denied the opportunity to use them. Obviously, the $1 15,029 spent 
by ETP could have been better spent elsewhere. 

• In-house trainers have a big impact 

We found that when in-house trainers performed training, the impact was greater. This is not 
surprising because the training can be better individualized to the company and because the 
expertise behind the training is retained within the company. It also communicates to the 
workers that management considers the training to be important. Interestingly, in our review 
of related research we found a study done with the Department of Labor survey data which 
also found that if an employer-provided the training directly, earnings rose 5%; whereas if a 
vendor provided the training increases were insignificant (Lengermann, 1996). Below, we 
discuss the main findings regarding “stand-alone” training. 

We have already discussed the very successful Basic Batteries training, which used in-house 
trainers who were trained by an outside consultant. Another example, Flow Pumps, relates 
how one company took the in-house trainer idea to the limit: the president of the company 
did some of the training. 



Flow Pumps 

Trained 190 of 320 workers 
ETP Costs: $215,880 
Estimated Benefits: 

■ 25% - 30% improvement in “Direct Labor Productivity” 

■ 75% reduction in scrap rates 



Flow Pumps is a company that makes high-end stainless steel pumps. The project was to 
train 190 workers in a variety of areas including production techniques, basic math, and SPC. 
Among the 190 were 24 managers and 24 supervisors who received management skills 
training. Initially the company used a few community college instructors for basic math 
training, but most training was ultimately provided by supervisors, engineers, and workers 
who had received special train-the-trainer training. The president, an engineer by trade, 
taught basic math to a number of trainees. 

The president sent us a graph showing “direct labor productivity.” This important measure 
varied between 75% and 80%. Before training and since then it has varied between 80% and 
120% hovering mostly around 100%. Overall, the president believes “the teams have done a 
good job on what they can control.” He claims there has been a 75% reduction in scrap rates, 
from 10% to 2.5%. He believes that return and warranty work has declined dramatically. In 
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addition, turn over and absenteeism have declined significantly, but no hard numbers were 
available. About half the trainees (47%) reported that overall training led to a “substantial” 
or “major” increase in productivity. They further report that SPC and Production Techniques 
had the biggest impact on productivity. Trainees also agree that the training led to positive 
changes in the work environment. 

• Training may be required by an outside party 

Some companies were required by major customers to train their workers.' So, in order to 
survive, they had to train. This brings up the issue that training may have little noticeable 
impact beyond the fact that it allows the company to retain a major customer or simply stay 
in business, and illustrates that benefits to training are not always easily quantifiable. 



TAC Aviation 
92 workers trained out of 150 
ETP Costs: $50,150 
Estimated Benefits: 

■ $25,000 per year 

■ Retained a key customer 

■ Cost-benefit ROI: about 50% per year over and above survival 



TAC Aviation is a small, privately-owned, union shop which is involved in sheet metal 
fabrication for the aerospace industry. TAC trained 92 of its workers but was not 
forthcoming with financial data on the impact of training. The workers commented that the 
TQM was beneficial and improved communications with managers and engineers. However, 
there was little attempt to keep records of the impact of training. The head of Human 
Resources said that TQM-inspired ideas are providing direct savings in the neighborhood of 
$25,000 per year. She also commented that some trainees left for higher paying jobs due to 
their SPC and TQM training which was required by Boeing, a major customer. Compared 
with the $50,150 spent, the benefits from ETP funding were paid back in two years if the 
estimate is correct. An additional impact would be the higher wages earned by trainees who 
left for higher pay elsewhere. 
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VI Consortia and Training Agency Contracts 



Basic Types of Training Contracts 

ETP enrolled 57,485 trainees in contracts that completed in fiscal 1994-95 and 1995-96, the 
study period. While the training contracts differed in a variety of ways, there were three 
basic types of training contracts: 

^ stand-alone contracts, 

S consortia contracts, and 
^ training agency contracts. 

The most common type of ETP training contract was the “stand-alone” contract in which 
ETP contracted with one company to provide training for its workers. In these contracts the 
training may be provided by employees of the company (in-house trainers) or by outside 
trainers working under a subcontract (subcontractors or consultants). About 60% of the 
trainees were trained under stand-alone contracts during the study period of 1994-96. 

The “consortia” contracts accounted for about 10% of the trainees and were the smallest 
group. Consortia may be large businesses training their own workers and some from their 
suppliers, Private Industry Councils (PICs) training workers for a particular group of 
businesses, or industry associations training workers for a particular group of their members. 
Consortia contracts are similar to stand-alone contractors in that there is a given set of 
workers who will be served but, unlike the stand-alone contractors, these workers are 
employed in more than one business. 

Training agency contracts were the second largest group of contracts and accounted for an 
increasing percentage of trainees during the study period. Training agencies served just over 
20% of trainees in fiscal year 1994-95 and just over 30% in fiscal 1995-96. Training 
agencies are often community colleges, private training organizations, or industry 
associations that contract to provide a certain type of training to workers in ETP eligible 
businesses. These training agencies market ETP funded training to nearby businesses and 
bear the risk of recruited trainees not completing the required placement and retention on the 
job (90 days), in which case the trainer would not be paid for the training. 

As Figure 7 shows, ETP is moving away from “stand-alone” projects and towards “training 
agency” projects. In one year, ETP moved from serving 64% of trainees through “stand- 
alone projects, to about 58%. Conversely, the percent of trainees served by “training 
agency” contracts rose from 23% to almost 32%. 
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Figure 7 

Percentage Distribution of Enrollees by Type of Contract 




Both consortia and training agencies help ETP to serve smaller businesses that would not 
find it worthwhile to undertake the cost of a stand-alone ETP contract. In this way, training 
agencies and consortia are very similar. The defining difference is that consortia serve a 
specific group of businesses, while training agency contractors market their training to a wide 
array of businesses, limited only by the businesses’ interest in the particular type of training 
offered and the convenience of the training site. 

We visited all three types of training contractors during our fieldwork, apportioning our visits 
in a rough approximation of their size. In addition to these three types of contracts, we 
discerned two types of training agency contracts in our field visits. 

S Industry Specific Training Agency Contracts: These contracts involve training in 
skills used by a single industry or narrow cluster of industries, for example, CNC 
machining. 

^ Generic Skills Training Agency Contracts: These contracts involve training in 
skills used in almost all industries, for example, word processing. 

Given the shifting pattern of contracts we believe it is important to look at each type of 
contract in some detail. We use our training model to conduct this analysis. The following 
matrix sums up the key characteristics and their implications. 
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Potential Training Effectiveness by Contract Type 
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The type of contract affects the potential impact and effectiveness of ETP training. The 
training model we introduced earlier (and show again here) identifies the two critical areas of 
“quality of training” (training design X and quality of training delivered), and “management 
reinforcement of training.” Together the quality of training and its reinforcement determine 
how much of a company’s potential training gains is actually realized. Both the quality of 
the training design and the extent of management reinforcement are likely to vary 
systematically with the different types of contracts because of the nature of those contracts. 
Stand-alone and consortia contracts are more likely to help companies achieve their full 
potential gains because they do not have the limits inherent in training agency contracts. 
Training agency contracts, particularly generic training ones, have structural constraints on 
their ability to deliver the potential gains from training. 

^ Stand-alone and consortia contracts 

Stand-alone and consortia contracts can achieve all the potential gain inherent in the 
companies they serve. Stand-alone contractors have great flexibility to shape the training 
they provide to the unique characteristics of the companies and trainees. By the same token, 
consortia contractors have virtually the same flexibility in their approach since, ideally, the ’ 
contractor assembles a group of businesses that share the same training requirements. For 
example, one large consortia contract trains auto mechanics from many different small 
service centers to use computerized diagnostic equipment. Essentially all these mechanics 
need the same skills and will use them in similar work environments. In this case, training 
can be customized almost as much as it could in stand-alone projects. Projects are 
customized by taking into account the initial skill level of the trainees and then providing the 
specific skills that they will need to perform on the job. Similarly, the management can 
interact closely with trainers and make sure that the already customized training is reinforced 
on the job. 

Clearly, both stand-alone and consortia curricula can be poorly designed, with poor training 
delivery, and little management reinforcement, and consequently deliver a low impact. 
Another potential pitfall is putting together a consortium where the training requirements are 
different for the various companies in the consortium. In this case, it would be difficult to 
customize the training for the trainees when there is wide variation in the trainees’ initial skill 
levels and/or in the skills they are expected to have after training. Obviously, the consortium 
should be constructed so that the initial skill and expected skills would be common among 
the consortium members. The point here is that both stand-alone and consortia contractors 
have the potential to provide excellent training design, delivery, and reinforcement, thus 
producing highly effective training. 
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'f Industry specific skills training agency contracts 

The ability of training agencies conducting industry-specific training to help companies 
achieve all of their potential gains is somewhat limited by the nature of the contract. These 
agencies provide a particular type of training to employees of a single-industry or a narrow 
cluster of industries. If trainees entering the program have fairly standard initial skill levels 
and standard skills are expected after training, then designing and reinforcing training will be 
relatively easy. However, to the extent that the initial skills vary widely among the trainees 
or that the tasks to be performed by the trainees after training are different, the training 
design will serve the trainees poorly. If the entry-level skills of the trainees differ and the 
contractor aims for some average level competency, then some of the trainees will not be 
able to master the instruction at the outset, while others will waste time being instructed in 
areas they already have mastered. Also, it will be difficult for trainers to select training 
exercises that are germane to all the trainees’ tasks. This is particularly problematic in the 
use of SOST. It is interesting to note how one industry-specific contract we visited, which 
taught CNC machining, dealt with this problem. First, to ensure uniform skills upon entry, 
they began the program with a math unit to bring all trainees math skills up to an acceptable 
level. Next, it decided not to use SOST. Rather, it provided extensive lab time on the 
training center’s machines so that they could be sure class skills were reinforced. 

Management reinforcement of training also can be difficult in industry-specific training 
agency contracts. The training is designed and implemented by the training agency, but the 
on-the-job reinforcement of the skills is in the hands of the various companies where the 
trainees work. There are bound to be variations in the extent to which different companies 
reinforce the training, especially since the management of these companies does not 
necessarily know the exact nature of the training. For example, in the same CNC training 
program, we found that some companies kept careful track of what trainees were learning 
and gave them more challenging tasks as they progressed. Other managers had very limited 
awareness of the learning that was going on and made no attempt to upgrade the trainee’s job 
to match his or her skill. Again, if the training involves skills that are fairly standard 
throughout the industry, companies will know what the skills are and can more easily 
reinforce the training on the job. But if the tasks the companies expect the trainees to 
perform vary widely, then many trainees will waste time being instructed in skills they will 
not use, and which management cannot reinforce. A final barrier to close ties between 
training and the employer is the fact that much of this training takes place off-the-clock. 

Since the employer is not paying for the trainee’s time, he or she is less motivated to ensure 
that new skills are put to use promptly to boost productivity. We want to note that despite 
these barriers, the opportunity for close ties between trainers and employers are better in this 
case than in the generic skills contracts, because close ties between the training agency and 
the industry tend to develop over time, particularly if the training agency is formally 
affiliated with the industry, which often appears to be the case. 

In summary, industry-specific training by training agencies will have a higher impact on 
companies, and be more effective for workers, if the trainees’ entry skills are relatively 
standard and the tasks they are expected to perform on completion are uniform. Also, the 
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impact is more significant the closer the ties between trainers and employers. The potential 
gains realized under these contracts are somewhat limited primarily because of the 
management reinforcement issue, but the difficulty of designing an appropriate training 
curriculum for a diverse group of trainees also poses a barrier. 

S Generic skills training agency contracts 

Generic skills training agency contract programs are less able to customize their curriculum 
for individual employers, and it is less likely there will be effective management 
reinforcement of training. By their nature, generic skills are applied across a large number of 
industries and one expects to find many different types of applications of the skills among 
those industries. The initial skill levels of the trainees are also widely varied which makes it 
difficult to customize instruction. For example, we interviewed one instructor who taught 
office automation in a generic skills contract. He pointed out that some of his students were 
computer literate and knew the keyboard well, while other students had never worked on a 
computer and didn’t know the keyboard at all. This made it very difficult to find an 
appropriate level at which to teach. Inevitably, appropriate starting points will be beyond the 
grasp of some and a waste of time for others. Classroom applications that are germane to 
some will not be for others. Very few trainees will use all of the skills covered in a generic 
skills program. In another case, we found several trainees in generic office automation 
training who were engineers or accounting clerks. They took the training to learn advanced 
spreadsheet applications, but to get to that section of the training they had to sit through 
many hours of word processing instruction which was of little value to them. 

Obviously, management’s on-the-job reinforcement of generic skills training will be difficult. 
Managers in the companies to which the workers return are distant from the trainers and 
unlikely to have intimate knowledge of the skills taught. This creates several impediments to 
effective reinforcement of the training. First, because the trainers and managers are in 
different organizations, the trainers have to coax the managers to reinforce the training. 
Secondly, the managers’ lack of knowledge of the training content makes it difficult to 
reinforce the training. Furthermore, the post-training job is unlikely to make use of all of the 
skills, which eliminates any possibility of reinforcing the unused skills. Generic skills 
training is also the situation where trainees are most likely to be attending on their own time 
which, as noted before, does not motivate employers to become involved. 

The training impact model implies that the ability of generic skills training to realize the full 
potential gains from training is limited. The limitations derive from the difficulty in 
designing training that is appropriate to workers from a wide range of industries with a wide 
variety of skills, to select training exercises relevant to most of the trainees, and to persuade 
the managers of the disparate companies to reinforce the training. 

Interestingly, one of the difficulties faced by the generic skills training agencies reduces the 
importance of ETP-funding of this type of training. Generic skills training agencies must 
find some skill areas that are useful to many industries and employees, and then design a 
curriculum appropriate to a large number of prospective trainees that work in various 
industries. To the extent that such skill areas exist, the market for training has already 
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recognized the demand for training in these areas, and training opportunities are widely 
available without ETP funding. This means that much of the ETP-funded training in generic 
skill areas may replace training that would have occurred anyway. 

Another phenomenon we observed in the field is that training agencies, which began by 
providing industry-specific or consortia training, are moving Quickly into generic training. 
There appears to be two reasons for this trend. First, the market for generic training is larger, 
since the skills taught stretch across many industries. Second, generic training is easier and 
less costly to deliver because it is standard and not customized. Thus instructors do not need 
specialized industry knowledge, and a single curriculum can be used for many classes. Our 
impression is that generic training is generally easier to manage and more profitable than 
specialized training. 

Generally, both the companies and the trainees thought that generic skills training was 
beneficial to the company and made the individual workers more productive. Of course, this 
means that the company and the individuals would likely seek out this type of training in the 
market if the ETP fiinded training did not exist. 

The training model implies that it will become increasingly difficult to achieve excellent 
training as we move from consortia training, to industry-specific skills training agencies, and 
finally, to generic skills training agencies. In the table below, we selected the trainees served 
by consortia and training agencies and then looked at the percent who rated different aspects 
of their training as excellent in Table 5. Since most of the consortia and training agency 
projects we visited were well-designed given the nature of the training that was undertaken, 
we believe that the relative percentages of excellent responses speaks to the inherent 
limitations on training effectiveness by the type of training provider. Overall, we were 
impressed by the quality of training some of the generic trainers were providing, given the 
challenges they faced. 
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Tables 

Trainees Rating of Training Quality and Amount Learned 
(Percent Reporting “Excellent”) 



Aspect of Training 


Percentage responding “excellent” by type 
of contract 






Training 


1 Agency 




Consortia 


Industry-Specific 


Generic 


Overall training quality 


61% 


60% 


37% 


Clear training objectives 


42% 


67% 


42% 


Usefulness of tropics 


40% 


53% 


32% 


Length of time on each tropic 


5% 


27% 


7% 


Quality of instructional materials 


50% 


64% 


40% 


Degree training customized to company 


36% 


15% 


8% 


Quality of instructors 


100% 


73% 


55% 


Effectiveness of SOST 


26% 


46% 


27% 


Ability of trainers to keep interest 


86% 


53% 


47% 


Right level for trainee 


50% 


53% 


23% 


How much of all training was actually learned 
(Percent reporting learned 60% or more) 


74% 


90% 


68% 



These trainee responses generally confirm the limitations on training effectiveness implied 
by the training impact model. 



• The Impact of Training Agency and Consortia Training on Companies 

In our fieldwork, we probed the companies’ managers for any indications they could provide 
of the overall impact of ETP training on the company. We combined these indications with 
what we could glean from standard financial sources in attempts to determine the impact of 
training on the companies involved. This proved particularly difficult for training that 
occurred in consortia and training agencies for a number of reasons associated with the size 
of the companies involved and the nature of training agency and consortia contracts. Also 
we only visited three companies in each training agency or consortia project when in fact the 
trained workers often form dozens of companies. 

What was considered a major advantage to training agencies and consortia in reaching small 
companies became a distinct disadvantage in assessing the impact of the training on those 
companies. Namely, smaller companies generally have less formal record-keeping systems 
and engage in less financial analysis and planning. This means that less insight, data, and 
analyses are at the fingertips of the managers who were queried about the impact. 
Furthermore, the smaller companies are less likely to show up in the financial databases that 
were searched for financial histories of the companies. This means that neither management 
nor the standard financial databases could provide much reliable information regarding the 
financial impact on the companies involved. 
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Even if financial data were available for the involved companies, the nature of the training 
agency and consortia programs mitigate against detecting the financial effects of the training 
on the company. Especially under training agency contracts, the training tends not to be 
done as a single event in calendar time. Rather, the companies partaking in this training tend 
to send a small contingent of their workers (1-3) at a time to the more or less ongoing 
training agencies sessions. This spreads the training of the employees over a number of years 
so that any financial impact is also spread over a number of years. The fact that they send a 
small contingent of workers at any one time also limits the impact of the training on the 
company at any one time. The absence of the “training event” that occurs in the large stand- 
alone ETP contracts makes the before and after financial comparisons impossible because 
there is no discrete before or after training period for these companies. 

With those cautions in mind, our fieldwork probing and financial analyses revealed a general 
pattern where the training impact on companies were more notable for those involved in 
specific industry training agency or consortia training. These impacts were detectable 
because the workers trained in industry-specific skills “made things,” so changes in scrap 
rates, shop capabilities, production time, and the like, were noticed and tracked. Consortia 
training generally involved intact work groups that did things in notably different ways after 
training, so the results of this training was also noticed by management. 

Generic training by training agencies, on the other hand, produced few detectable impacts on 
the companies. The small contingents of company workers who were trained tended to be in 
support roles where changes in productivity were not subject to convenient measures. In 
addition, these support activities are likely to be small relative to the overall company 
activity, which would also diminish any measurable impact of the training on the company. 

In one case where a significant number of company workers were trained on new software, 
the company was simply switching from one type of software to another, a change which did 
not particularly alter the trainees’ capabilities or productivity. 

In summary, we detected the greatest impact of training on companies that trained under 
consortia or industry-specific training contracts. The results of this training tended to be 
measurable and to be tracked (though not always precisely) by management. Training in the 
generic skills training agency contracts produced little discemable impact on the companies 
involved, possibly because of: the way the companies made use of the training availability 
(over several years); the small number of employees involved; their supportive roles; or, 
possibly because the training did not have much impact on the company’s capabilities or 
productivity. 
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Appendix A 



Trainee Evaluation Questionnaire 



California State University , Northridge 



ETP Impact Assessment Project 



Trainee Evaluation Questionnaire 

This questionnaire will help us get your opinion about the training that you received. The 
information will be used by the state agency, the Employment Training Panel to improve their 
program. Your answers are completely confidential . and will not be shown to your employer. 

1 . What is your current job title? 

2. How many years have you worked for the company? years 



3. How would you rate the training you received in the following areas? 
(Check box that indicates your opinion) 



Component 


Rating 1 




Excellent 


Good 


Fair 


Poor 


Clear Objectives 










Usefulness of topics covered for 
your job. 










Length of time on each topic. 










Quality of instructional materials 










Degree training was customized to 
your company 










Quality of instructors 










Effectiveness of SOST* in 
allowing you to practice the skills 
you learned 










Ability of trainers to keep you 
interested and motivated 










Degree to which training was at the 
right level for you, not too easy or 
not too hard 










Quality of training overall 











*Structured On Site Training 



4. The best thing about the training was: 
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5. The worst thing about training was: 



6. Training Effectiveness: Of the material presented, how much did you actually learn well 
enough to use? 



Skill or Knowledge: 


Mastery of Skills or Knowledge : . 


Trainees 
did not ; 

; r take v 


vjvir. 




Every 

Thing 


Most • r 

(60%+): 


About 

Half 

'■■■ ’ .. ..t. ,: 


Some I 

(<40%) 


None 


SPC 














TQM 














Production Techniques 














Manufacturing Resource Planning 














Management Skills 














Office Automation 














CAD/CAM 














All Training Together 















7. Use: How often do you use these skills? 



Skill: or Knowledge ^ 


Use of Skills or Knowledge 


Trainees 
did not 
take: 

•’ . I ..'"! - . 


■ r, W - ^ •: ' .•i.-r-. Z- ; VV--' 

. • * : >,-1 • +V • V l; 1 << ^ • s ; , , 1 ;/ v ' •- •' , 

AL-- r X, 

a;;- 


Every 

Day 

% \ II * « t ' r J] 


Once a -y 
week or % 
: more 


Less than 
once a week 


- Never: 


SPC 












TQM 












Production Techniques 












Manufacturing Resource Planning 












Management Skills 












Office Automation 












CAD/CAM 












All Training Together 
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8. What was done to encourage you to use your new skills and knowledge when you were 
back on the job? 



9. Did anything discourage you from using your new skills and knowledge back on the job? 



10. Impact: Has your productivity changed because of training? 



^ Skill or Knowledge ' 


Impact on Productivity 




Trainees 
did not 
• v’ take . 




Major 

increase 


Substantial 
: £- increase 


• . j Small : 
increase 


• No 
increa 

■' ;• 1 . 

•• V.:: : : r • 


ise: 

m 


SPC 












TQM 












Production Techniques 












Manufacturing Resource Planning 












Management Skills 












Office Automation 












CAD/CAM 












All Training Together 













1 1 . Please give an example how the training has or has not changed your productivity. 
(Example: “I can set up my machine in half as much time.”) 
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1 2. Think about how things have changed at your company since you took the ETP training. 
Then read the statements below and rate if you agree or disagree. 



>• .•> . v" 
. : ", "... 


Strongly 
Agree • 


Agree! 


;Disagree5 


!$tfprigly 

^Disagree 


No 

Opinion 


Communication between supervisors and 
workers is better because of the training. 












I feel less stress on the job because of 
what I learned in the training. 












Workers are treated better because of the 
training. 












Relations between workers are better 
because of the training. 












I feel more motivated and involved at 
work than before the training. 












I am more optimistic about my future in 
the company because of my training. 












I have a more positive attitude about the 
company because of the training. 













13. Do you have any additional comments you would like to make about the ETP training? 



Thank you for your help. 
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Supervisor Evaluation Questionnaire 

This questionnaire will help us get your opinion about the training that you received. The 
information will be used by the state agency, the Employment Training Panel to improve their 
program. Your answers are compl etely confidential and will not be shown to your employer. 

1 . Of the workers you supervise, what percent participated in the ETP Training? 
% 

2. Did you also receive training? Yes No 



3. Did you serve as trainer in the training? Yes No 



4. To what degree were you involved in planning the training? (circle) 

All parts Some parts A few parts Not involved in planning 



5. How would you rate the following components of the training? 



'i' • '4 % ;,i ' •. Component 






Excellent v 


Good 




i: Poor 


Clarity of training objectives. 










Usefulness of topics to trainees’ 
jobs 










Length of time spent on each topic 










Quality of instructional materials 










Degree training was customized to 
your company 










Quality of instructors 










Ability of training to keep trainees 
interested and motivated 










Effectiveness of Structured On-site 
Training 










Degree to which training was at an 
appropriate level for trainees 










Training Overall 
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6. Was training worth the time it took away from production? (circle) 

Definitely Yes Not Sure Definitely Not 

Why? 



7. The best thing about the training was: 



8. The worst thing about the training was:. 



Training Effectiveness: Of the material presented, to what degree did the trainees learn 
what was taught? 



Skill or Knowledge 1 : r- 


Mastery of Skills or Knowledee ; r > ■ 


n Trainees 
/ V did hot 
take , 




y ■: • : yiv /V-;; ; ; . . / - 


Every 

Thing 


Most 

(60%+); 


About 
r Half 


Some 

:(^40%| 


None 


SPC 














TQM 














Production Techniques 














Manufacturing Resource Planning 














Management Skills 














Office Automation 














CAD/CAM 














All Training Together 
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10. How often have trainees been able to use the new skills and knowledge they got from 
training on the job? 



filCilCSkiU : or Knowledge? 


■■■ Use of Skills or Knowledge ■ Ti: 


Trainees 
did not ; 

? take '.(t 


• .7 If. C' * • ! > ' / -'C'-' 1 '' \ “K ‘ 


Every 


Once a week 
f or more ti 


<Once a 
1 week 


Never ; 


SPC 












TQM 












Production Techniques 












Manufacturing Resource Planning 












Management Skills 












Office Automation 












CAD/CAM 












All Training Together 













1 1 . What, if anything, has encouraged trainees to use their new skills and knowledge? 



What, if anything, has discouraged trainees to use their new skills and knowledge? 



BEST COPY AVAILABLE 

o 

ERIC 



81 



3 



California State University Northridge 



ETP Impact Assessment Project 



What impact has training in each skill or knowledge area had on the productivity of the 
area you supervise? 



; Skill or Knowledge 


Impact on Productivity 


Trainees 
did not 
take 






> ; Major 
Increase 


Substantial: 
i -Increase 


; Small 
Increase 


No . 
Increase 


SPC 












TQM 












Production Techniques 












Manufacturing Resource Planning 












Management Skills 












Office Automation 












CAD/CAM 












All Training Together 












AutoCAD 













Please give one or more examples of how the training has or has not had an impact on the 
productivity of the workers you supervise. (Examples: “Our scrap rate declined from 
15% to 8% after training;” or “Work absenteeism fell from 8% to 2%;” or, “We expected 
a decline in our defect rate and it didn’t happen.”) 



1 5. Comparing the period before the ETP training with now, has the amount of training 
provided production workers changed? (circle) 

A lot more training More training No change Less training 

16. Comparing the period before the ETP training with now, how has the company’s attitude 
toward training changed? (circle) 

A lot more interest & A little more interest No change Less interest & 

support & support support 
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17. Comparing the period before the ETP training with now, how have production workers’ 
attitudes toward training changed? (circle) 

A lot more interest & A little more interest No change Less interest & 

support & support support 



18. Think about how things have changed at your company since you took the ETP training. 
Then read the statements below and rate if you agree or disagree. 



| Statement 


Strongly 

Agree 


Agree 


Disagree 


.'Strongly 

Disagree 


No 

Opinion 


Communication between supervisors and 
workers is better because of the training. 












I feel less stress on the job because of 
what I learned in the training. 












Workers are treated better because of the 
training. 












Relations between workers are better 
because of the training. 












I feel more motivated and involved at 
work than before the training. 












I am more optimistic about my future in 
the company because of my training. 












I have a more positive attitude about the 
company because of the training. 













Do you have any additional comments you would like to make about the ETP training? 



Thank you for your help 
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